How to use API with payload

[stif]

Hi folks,
I am able to use the API in order to fetch information, but i am failing to add new items and i think my problem is how to add the payload to POST requests..

Here is my Code, wich just gets a `{'result': 'failed'}` back, but i have no idea why it fails. do i need another Content-Type Header?

Code: [Select]

import requests
import json
from requests.auth import HTTPBasicAuth
import os

# OPNsense API
#https://opnsense.local/api/<module>/<controller>/<command>/[<param1>/[<param2>/...]]

# Globale Konfiguration

API_KEY = os.getenv('API_KEY')
API_SECRET = os.getenv('API_SECRET')
OPNSENSE_URL = "https://10.1.1.11:8443"

INIT_DATA = {
    'vlan': {
        "vlanTag": "211",
        "interface": "lagg0",
        "description": "newVLAN"
    }
}

def send_request(url, data=None, method='POST'):
    headers = {}
    auth = HTTPBasicAuth(API_KEY, API_SECRET)
    verify_ssl = False  # Für Produktionscode sollten Sie SSL-Zertifikatsprüfung aktivieren

    if method in ['POST', 'PUT', 'PATCH']:
        headers['Content-Type'] = 'application/xml'
        response = requests.post(url, headers=headers, data=json.dumps(data), auth=auth, verify=verify_ssl)
    else:
        response = requests.get(url, headers=headers, auth=auth, verify=verify_ssl)

    if response.ok:
        return response.json()
    else:
        response.raise_for_status()

def add_vlan(vlan_data):
    """Adds a new VLAN."""
    url = f"{OPNSENSE_URL}/api/interfaces/vlan_settings/addItem"
   
    # Erstellen der XML-Payload mit den übergebenen Werten
    xml_payload = f"""
    <vlan>
        <if>{vlan_data['interface']}</if>
        <tag>{vlan_data['vlanTag']}</tag>
        <pcp>0</pcp>
        <proto></proto>
        <descr>{vlan_data['description']}</descr>
        <vlanif>vlan0{vlan_data['vlanTag']}</vlanif>
    </vlan>
    """
    print(xml_payload)
    # Senden des Requests mit der XML-Payload
    response = send_request(url, xml_payload)
    return response

# Main function to organize operations
def main():
    response = add_vlan(INIT_DATA['vlan'])
    print (response)

if __name__ == "__main__":
    main()


[stif]

OK i found this post: https://forum.opnsense.org/index.php?topic=37399.msg184808#msg184808 suggesting "record" a request by using the OPNsense Web interface and the Developer Tools..
and after seeing a correct POST request i could see the payload is JSON instead of XML..

This was not clear to me after reading:
https://docs.opnsense.org/development/api/core/interfaces.html and
https://github.com/opnsense/core/blob/master/src/opnsense/mvc/app/models/OPNsense/Interfaces/Vlan.xml

Maybe the docs could be more precise about it. I got it working now:

Code: [Select]

def add_vlan(vlan_data):
    """Adds a new VLAN."""
    url = f"{OPNSENSE_URL}/api/interfaces/vlan_settings/addItem"
   
    # Erstellen der XML-Payload mit den übergebenen Werten
    # eg {"vlan":{"vlanif":"vlan0211","if":"lagg0","tag":"211","pcp":"0","proto":"","descr":"Victor"}}
    payload = {
    'vlan': {
        'vlanif': 'vlan0' + vlan_data['vlanTag'],
        'if': vlan_data['interface'],
        'tag': vlan_data['vlanTag'],
        'pcp': '0',
        'proto': '',
        'descr': vlan_data['description']
        }
    }

    # Senden des Requests mit der JSON-Payload
    response = send_request(url, payload)
    return response

Edit: ok now i found "The body of the HTTP POST request and response is an ‘application/json’ object." in the first lines of the API reference  nevermind..

[Patrick M. Hausen]

[stif]

now i know how to use the API, i just found out i cannot use it in my scenaro 
I wanted to make a clean version of batch creating some stuff: https://forum.opnsense.org/index.php?topic=37125.0

I would like to use the API to create a new VLAN, create an Interface with it and configure DHCP, CARP, NAT and Firewall Rules for it.

But i do not find a way to assign the API created VLAN to an Interface??
Since the Web UI is also using interfaces_assign.php for it i guess it is not possible?

[stif]

i captured the "assign interface" request with developer tools and wanted to replicate this request but found 2 problems:

Code: [Select]

POST https://10.1.1.11:8443/interfaces_assign.php
payload:
[SomeToken]=[anotherToken]&action=&id=&lan=lagg0&opt2=vlan04&opt3=vlan05&opt4=vlan08&[....]&opt83=vlan0210&wan=ix0&opt1=ix1&if_add=vlan0211&new_entry_descr=Vlan Description&add_x=

1. problem: Anybody knows where to get [SomeToken] and [anotherToken]?

and in order to gather all interfaces details to put together the payload string i wanted to use the API /api/interfaces/overview/export but got the response:
{'message': 'controller OPNsense\\Interfaces\\Api\\OverviewController not found', 'status': 400}

2. problem: Why there is no OverviewController found?

I am using OPNsense Business Version 23.10.3 if it does matter

[stif]

I did the whole workflow with Web UI and captured all the requests. Unfortunately less than half of the steps necessary, are available via API:

Code: [Select]

(tokens are replaced by random strings the same length (32 chars))
export BASE_URL="https://10.1.1.11:8443"

# Add Vlan
curl -X POST \
    --header 'Content-Type: application/json' \
    --user "${API_KEY}:${API_SECRET}" \
    --insecure \
    --data-raw '{"vlan":{"vlanif":"vlan0211","if":"lagg0","tag":"211","pcp":"0","proto":"","descr":"Victor"}}' \
    "${BASE_URL}/api/interfaces/vlan_settings/addItem"

# Apply Vlan
curl -X POST \
    --header 'Content-Type: application/json' \
    --user "${API_KEY}:${API_SECRET}" \
    --insecure \
    --data-raw '{}' \
    "${BASE_URL}/api/interfaces/vlan_settings/reconfigure"
   
# Assign Interface
curl 'https://10.1.1.11:8443/interfaces_assign.php' \
-X POST \
-H 'Referer: https://10.1.1.11:8443/interfaces_assign.php'\
-H 'Content-Type: application/x-www-form-urlencoded' \
-H 'X-CSRFToken: 7UP69EdWfRuTGGsL8Vwravt3vqRtKhpc' \
-H 'X-Requested-With: XMLHttpRequest' \
-H 'Origin: https://10.1.1.11:8443' \
-H 'Cookie: PHPSESSID=cSvTEwesyHg57RRsHMkJP9VR064wb9jc' \
--insecure \
--data-raw 'J0Rmr2EzDOo6jQxl2tJVbWL9xEpVaZOq=7UP69EdWfRuTGGsL8Vwravt3vqRtKhpc&action=&id=&lan=lagg0&opt2=vlan04&opt3=vlan05&opt4=vlan08&opt5=vlan010&opt6=vlan011&opt7=vlan012&opt8=vlan013&opt9=vlan014&opt10=vlan015&opt11=vlan016&opt12=vlan017&opt13=vlan018&opt14=vlan019&opt15=vlan020&opt16=vlan022&opt17=vlan023&opt18=vlan024&opt19=vlan025&opt20=vlan026&opt21=vlan027&opt22=vlan028&opt23=vlan029&opt24=vlan030&opt25=vlan032&opt26=vlan033&opt27=vlan034&opt28=vlan035&opt29=vlan036&opt30=vlan037&opt31=vlan038&opt32=vlan039&opt33=vlan040&opt34=vlan041&opt35=vlan042&opt36=vlan043&opt37=vlan044&opt38=vlan045&opt39=vlan046&opt40=vlan047&opt41=vlan048&opt42=vlan049&opt43=vlan050&opt44=vlan051&opt45=vlan052&opt46=vlan053&opt47=vlan054&opt48=vlan055&opt49=vlan056&opt50=vlan057&opt51=vlan058&opt52=vlan059&opt53=vlan060&opt54=vlan061&opt55=vlan062&opt56=vlan066&opt57=vlan070&opt58=vlan080&opt59=vlan091&opt60=vlan092&opt61=vlan093&opt62=vlan094&opt63=vlan095&opt64=vlan096&opt65=vlan097&opt66=vlan098&opt67=vlan099&opt68=vlan0101&opt69=vlan0102&opt70=vlan0103&opt71=vlan0104&opt72=vlan0105&opt73=vlan0106&opt74=vlan0107&opt75=vlan0108&opt76=vlan0109&opt77=vlan0110&opt78=vlan0111&opt79=vlan0201&opt80=vlan0202&opt81=vlan0203&opt82=vlan0204&opt83=vlan0210&wan=ix0&opt1=ix1&if_add=vlan0211&new_entry_descr=V211_Victor&add_x='

# Select Interface
curl 'https://10.1.1.11:8443/interfaces.php?if=opt84' \
-H 'Referer: https://10.1.1.11:8443/interfaces_assign.php'
-H 'X-CSRFToken: 7UP69EdWfRuTGGsL8Vwravt3vqRtKhpc' \
-H 'X-Requested-With: XMLHttpRequest' \
-H 'Origin: https://10.1.1.11:8443' \
-H 'Cookie: PHPSESSID=cSvTEwesyHg57RRsHMkJP9VR064wb9jc' \
--insecure \

# Enable Interface
curl 'https://10.1.1.11:8443/interfaces.php?if=opt84' \
-X POST \
-H 'Referer: https://10.1.1.11:8443/interfaces.php?if=opt84' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-H 'X-CSRFToken: 7UP69EdWfRuTGGsL8Vwravt3vqRtKhpc' \
-H 'X-Requested-With: XMLHttpRequest' \
-H 'Origin: https://10.1.1.11:8443' \
-H 'Cookie: PHPSESSID=cSvTEwesyHg57RRsHMkJP9VR064wb9jc' \
--insecure \
--data-raw 'J0Rmr2EzDOo6jQxl2tJVbWL9xEpVaZOq=7UP69EdWfRuTGGsL8Vwravt3vqRtKhpc&enable=yes&descr=V211_Victor&type=staticv4&type6=none&spoofmac=&mtu=&mss=&ipaddr=10.20.211.251&subnet=24&gateway=none&name=V211_Victor_GWv4&gatewayip=&gatewaydescr=&alias-address=&alias-subnet=32&dhcprejectfrom=&dhcphostname=&dhcpoverridemtu=yes&dhcpvlanprio=&adv_dhcp_pt_timeout=&adv_dhcp_pt_retry=&adv_dhcp_pt_select_timeout=&adv_dhcp_pt_reboot=&adv_dhcp_pt_backoff_cutoff=&adv_dhcp_pt_initial_interval=&adv_dhcp_pt_values=SavedCfg&adv_dhcp_send_options=&adv_dhcp_request_options=&adv_dhcp_required_options=&adv_dhcp_option_modifiers=&adv_dhcp_config_file_override_path=&ports=%2Fdev%2Fcuau0&country=&provider_list=&providerplan=&username=&password=&phone=&apn=&pppoe_username=&pppoe_password=&provider=&pppoe_hostuniq=&pppoe_idletimeout=&pptp_username=&pptp_password=&localip=&pptp_subnet=31&pptp_remote=&pptp_idletimeout=&ipaddrv6=&subnetv6=128&gatewayv6=none&namev6=V211_Victor_GWv6&gatewayipv6=&gatewaydescrv6=&dhcp6-ia-pd-len=0&dhcp6vlanprio=&adv_dhcp6_interface_statement_send_options=&adv_dhcp6_interface_statement_request_options=&adv_dhcp6_interface_statement_script=&adv_dhcp6_id_assoc_statement_address_id=&adv_dhcp6_id_assoc_statement_address=&adv_dhcp6_id_assoc_statement_address_pltime=&adv_dhcp6_id_assoc_statement_address_vltime=&adv_dhcp6_id_assoc_statement_prefix_id=&adv_dhcp6_id_assoc_statement_prefix=&adv_dhcp6_id_assoc_statement_prefix_pltime=&adv_dhcp6_id_assoc_statement_prefix_vltime=&adv_dhcp6_prefix_interface_statement_sla_len=&adv_dhcp6_authentication_statement_authname=&adv_dhcp6_authentication_statement_protocol=&adv_dhcp6_authentication_statement_algorithm=&adv_dhcp6_authentication_statement_rdm=&adv_dhcp6_key_info_statement_keyname=&adv_dhcp6_key_info_statement_realm=&adv_dhcp6_key_info_statement_keyid=&adv_dhcp6_key_info_statement_secret=&adv_dhcp6_key_info_statement_expire=&adv_dhcp6_config_file_override_path=&prefix-6rd=&gateway-6rd=&prefix-6rd-v4plen=0&prefix-6rd-v4addr=&track6-prefix-id--hex=0&Submit=Save&if=opt84&ptpid=0'

# Apply Interface
curl 'https://10.1.1.11:8443/interfaces.php?if=opt84' \
-X POST \
-H 'Referer: https://10.1.1.11:8443/interfaces.php?if=opt84' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-H 'X-CSRFToken: 7UP69EdWfRuTGGsL8Vwravt3vqRtKhpc' \
-H 'X-Requested-With: XMLHttpRequest' \
-H 'Origin: https://10.1.1.11:8443' \
-H 'Cookie: PHPSESSID=cSvTEwesyHg57RRsHMkJP9VR064wb9jc' \
--insecure \
--data-raw 'J0Rmr2EzDOo6jQxl2tJVbWL9xEpVaZOq=7UP69EdWfRuTGGsL8Vwravt3vqRtKhpc&apply=Apply+changes&if=opt84'

# Repeat all Steps for Firewall2
* add Vlan
* apply Vlan
* assign Interface
* select Interface
* enable Interface
* apply Interface


# Get Group info
curl 'https://10.1.1.11:8443/api/firewall/group/searchItem' \
-X POST \
-H 'Referer: https://10.1.1.11:8443/ui/firewall/group' \
-H 'Content-Type: application/json;charset=utf-8' \
--user "${API_KEY}:${API_SECRET}" \
--insecure \
--data-raw '{"current":1,"rowCount":7,"sort":{},"searchPhrase":""}'

curl 'https://10.1.1.11:8443/api/firewall/group/getItem/ed7100a2-e802-4a87-821a-a61801e088f9' \
-H 'Referer: https://10.1.1.11:8443/ui/firewall/group' \
-H 'Content-Type: application/json'
--user "${API_KEY}:${API_SECRET}" \
--insecure \

# Add Interface to Group
curl 'https://10.1.1.11:8443/api/firewall/group/setItem/ed7100a2-e802-4a87-821a-a61801e088f9' \
-X POST \
-H 'Referer: https://10.1.1.11:8443/ui/firewall/group' \
-H 'Content-Type: application/json' \
--user "${API_KEY}:${API_SECRET}" \
--insecure \
--data-raw '{"group":{"ifname":"dhw_companies","members":"opt56,opt57,opt58,opt59,opt60,opt61,opt62,opt63,opt64,opt65,opt66,opt67,opt68,opt69,opt70,opt71,opt72,opt73,opt74,opt75,opt76,opt77,opt78,opt79,opt80,opt81,opt82,opt83,opt84","nogroup":"0","sequence":"20","descr":"all Company VLAN interfaces"}}'

# Add CARP IP
curl 'https://10.1.1.11:8443/api/interfaces/vip_settings/addItem/' \
-X POST \
-H 'Referer: https://10.1.1.11:8443/ui/interfaces/vip' \
-H 'Content-Type: application/json' \
--user "${API_KEY}:${API_SECRET}" \
--insecure \
--data-raw $'{"vip":{"mode":"carp","interface":"opt84","network":"10.20.211.254/24","gateway":"","noexpand":"0","nobind":"0","password":"cve7eYe3sOGifhYwz0Zc","vhid":"211","advbase":"1","advskew":"0","descr":"Victor K\xf6ssl"}}'

# Add outbound NAT
curl 'https://10.1.1.11:8443/firewall_nat_out_edit.php?dup=82' \
-X POST \
-H 'Referer: https://10.1.1.11:8443/firewall_nat_out_edit.php?dup=82' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-H 'X-CSRFToken: 7UP69EdWfRuTGGsL8Vwravt3vqRtKhpc' \
-H 'X-Requested-With: XMLHttpRequest' \
-H 'Origin: https://10.1.1.11:8443' \
-H 'Cookie: PHPSESSID=cSvTEwesyHg57RRsHMkJP9VR064wb9jc' \
--insecure \
--data-raw 'J0Rmr2EzDOo6jQxl2tJVbWL9xEpVaZOq=7UP69EdWfRuTGGsL8Vwravt3vqRtKhpc&interface=wan&ipprotocol=inet&protocol=any&source=opt84&sourceport=&destination=any&dstport=&targetip=84.242.14.75&natport=&poolopts=&tag=&tagged=&descr=Victor+K%C3%B6ssl&Submit=Save&after=82'

# Add Firewall Rule (allow own VLAN)
curl 'https://10.1.1.11:8443/firewall_rules_edit.php?if=opt84' \
-X POST
-H 'Referer: https://10.1.1.11:8443/firewall_rules_edit.php?if=opt84' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-H 'X-CSRFToken: 7UP69EdWfRuTGGsL8Vwravt3vqRtKhpc' \
-H 'X-Requested-With: XMLHttpRequest' \
-H 'Origin: https://10.1.1.11:8443' \
-H 'Cookie: PHPSESSID=cSvTEwesyHg57RRsHMkJP9VR064wb9jc' \
--insecure \
--data-raw 'J0Rmr2EzDOo6jQxl2tJVbWL9xEpVaZOq=7UP69EdWfRuTGGsL8Vwravt3vqRtKhpc&id=&after=&floating=&type=pass&quick=yes&interface=opt84&direction=in&ipprotocol=inet&protocol=any&icmptype=&icmp6-type=&src=opt84&dst=opt84&descr=+Allow+own+VLAN+network+&sched=&gateway=&reply-to=&set-prio=&set-prio-low=&prio=&tos=&tag=&tagged=&max=&max-src-nodes=&max-src-conn=&max-src-states=&max-src-conn-rate=&max-src-conn-rates=&overload=virusprot&statetimeout=&adaptivestart=&adaptiveend=&os=&statetype=keep+state&state-policy=&Submit=Save'

# Apply Firewall Rule
curl 'https://10.1.1.11:8443/firewall_rules.php?if=opt84' \
-X POST \
-H 'Referer: https://10.1.1.11:8443/firewall_rules.php?if=opt84' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-H 'X-CSRFToken: 7UP69EdWfRuTGGsL8Vwravt3vqRtKhpc' \
-H 'X-Requested-With: XMLHttpRequest' \
-H 'Origin: https://10.1.1.11:8443' \
-H 'Cookie: PHPSESSID=cSvTEwesyHg57RRsHMkJP9VR064wb9jc' \
--insecure \
--data-raw 'J0Rmr2EzDOo6jQxl2tJVbWL9xEpVaZOq=7UP69EdWfRuTGGsL8Vwravt3vqRtKhpc&id=&act=apply'

# Enable DHCP
curl 'https://10.1.1.11:8443/services_dhcp.php?if=opt84'
-X POST
-H 'Referer: https://10.1.1.11:8443/services_dhcp.php?if=opt84'
-H 'Content-Type: application/x-www-form-urlencoded'
-H 'X-CSRFToken: 7UP69EdWfRuTGGsL8Vwravt3vqRtKhpc' \
-H 'X-Requested-With: XMLHttpRequest' \
-H 'Origin: https://10.1.1.11:8443' \
-H 'Cookie: PHPSESSID=cSvTEwesyHg57RRsHMkJP9VR064wb9jc' \
--insecure \
--data-raw 'J0Rmr2EzDOo6jQxl2tJVbWL9xEpVaZOq=7UP69EdWfRuTGGsL8Vwravt3vqRtKhpc&enable=yes&range_from=10.20.211.1&range_to=10.20.211.100&wins1=&wins2=&dns1=10.20.211.254&dns2=&gateway=10.20.211.254&domain=&domainsearchlist=&defaultleasetime=&maxleasetime=&minsecs=&interface_mtu=&failover_peerip=10.20.211.252&failover_split=&ddnsdomain=&ddnsdomainprimary=&ddnsdomainkeyname=&ddnsdomainkey=&ddnsdomainalgorithm=hmac-md5&mac_allow=&mac_deny=&ntp1=&ntp2=&tftp=&bootfilename=&ldap=&nextserver=&filename=&filename32=&filename64=&filename32arm=&filename64arm=&filenameipxe=&rootpath=&omapiport=&omapialgorithm=&omapikey=&numberoptions_number%5B%5D=&numberoptions_type%5B%5D=text&numberoptions_value%5B%5D=&if=opt84&submit=Save'

# Sync With HA
curl 'https://10.1.1.11:8443/status_habackup.php' \
-X POST \
-H 'Referer: https://10.1.1.11:8443/status_habackup.php' \
-H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' \
-H 'X-CSRFToken: 7UP69EdWfRuTGGsL8Vwravt3vqRtKhpc' \
-H 'X-Requested-With: XMLHttpRequest' \
-H 'Origin: https://10.1.1.11:8443' \
-H 'Cookie: PHPSESSID=cSvTEwesyHg57RRsHMkJP9VR064wb9jc' \
--insecure \
--data-raw 'action=exec_sync'

curl 'https://10.1.1.11:8443/status_habackup.php' \
-X POST \
-H 'Referer: https://10.1.1.11:8443/status_habackup.php' \
-H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' \
-H 'X-CSRFToken: 7UP69EdWfRuTGGsL8Vwravt3vqRtKhpc' \
-H 'X-Requested-With: XMLHttpRequest' \
-H 'Origin: https://10.1.1.11:8443' \
-H 'Cookie: PHPSESSID=cSvTEwesyHg57RRsHMkJP9VR064wb9jc' \
--insecure \
--data-raw 'action=reload_templates'


I dont see a way to automate this steps with current state of the API.
Guess injecting xml parts into the config is the only practical way for my usecase..

But i am curios: the payload for non API requests alwas start with

Code: [Select]

--data-raw 'J0Rmr2EzDOo6jQxl2tJVbWL9xEpVaZOq=7UP69EdWfRuTGGsL8Vwravt3vqRtKhpcThe second part after "=" is the same as the X-CSRFToken, but what is the first part?
And how to get the Unknown-, the CSRF- and the PHPSession-Token without sniffing the webui requests?

[franco]

I wouldn't bother messing with URLs ending in ".php" because these are static pages without proper API support.

API actually has /api/ in the URL and the GUI parts /ui/


Cheers,
Franco

[stif]

API actually has /api/ in the URL and the GUI parts /ui/

And the GUI parts /ui/ itself uses the API, at least this is what i captured (all curl requests in my last post not ending in .php..)

Are there any plans to extend the API coverage? If yes is there a rough time frame?

And just for Info the OverviewController.php and NeighborSettingsController.php are not working in OPNsense 23.10.3:

Code: [Select]

{'message': 'controller OPNsense\\Interfaces\\Api\\NeighborSettingsController not found', 'status': 400}
{'message': 'controller OPNsense\\Interfaces\\Api\\OverviewController not found', 'status': 400}

Cheers,
Stefan

[Monviech (Cedrik)]

You can check the roadmap.

https://opnsense.org/about/road-map/

The things where it says "migrate to MVC" are the parts where the API works next. For example one of the biggest latest changes was the rewrite of System Trust.

So it's all coming, one by one, just takes time.