I am a noob in creating programs, however i’m seeking for learning in order to automate repetitive tasks. Hope you guys could help me.
The problem I’m facing here is that Im not beeing able to properlly execute the pool.dataset.create endpoint in order to automatically create a datasets. My final goal here is to create a program that allows me to create predefined datasets and subdatasets based on a master dataset name, because in my job the folders and subfolders need to follow a specif pattern, however I am stuck in the begging where Im trying to create a single dataset to test if the API is working.
I am writing the code in python. Have a look here:
functions.py →
import re
import websocket
import json
import ssl
# Function to validate ip addresses
def validate_ip_address(ip):
# Regular expression to validate an IP address
ip_pattern = r"^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})$"
# Check if the provided IP matches the pattern
match = re.match(ip_pattern, ip)
if match:
# Check if each octet is within the valid range (0-255)
for group in match.groups():
if not (0 <= int(group) <= 255):
return False
return True
else:
return False
# Function to connect and authenticate
def connect_and_authenticate(websocket_url):
# Configure SSL context to trust the server's certificate
sslopt_loose = {"cert_reqs": ssl.CERT_NONE}
# Connect to the WebSocket endpoint
ws = websocket.create_connection(websocket_url, sslopt=sslopt_loose)
# Connection Message
connect_message = {
"msg": "connect",
"version": "1",
"support": ["1"]
}
# Connect to server using the Connection Message
ws.send(json.dumps(connect_message))
# Receive connection response
response = json.loads(ws.recv())
# Check if the connection was successful
if response.get("msg") == "connected":
print("Connection established successfully!")
# Authenticate
username_userinput = "API_user"
password_userinput = "****"
auth_message = {
"id": "auth_request",
"msg": "method",
"method": "auth.login",
"params": [username_userinput, password_userinput]
}
ws.send(json.dumps(auth_message))
# Receive authentication response
auth_response = json.loads(ws.recv())
# Check if authentication was successful
if auth_response.get("result") is True:
print(f"Auth_response: {auth_response.get("result")}. Authentication successful!")
else:
print(f"Auth_response: {auth_response.get("result")}. Authentication failed.")
else:
print("Failed to establish connection.")
# Function to close the connection
def close_connection(websocket_url):
# Configure SSL context to trust the server's certificate
sslopt_loose = {"cert_reqs": ssl.CERT_NONE}
# Connect to the WebSocket endpoint
ws = websocket.create_connection(websocket_url, sslopt=sslopt_loose)
ws.close()
print("Connection closed successfully!")
# Function to create datasets
def create_dataset(websocket_url):
# Configure SSL context to trust the server's certificate
sslopt_loose = {"cert_reqs": ssl.CERT_NONE}
# Connect to the WebSocket endpoint
ws = websocket.create_connection(websocket_url, sslopt=sslopt_loose)
# Dataset Creation Message
create_dataset_message = {
"id": "6841f242-840a-11e6-a437-00e04d680384",
"msg": "method",
"method": "pool.dataset.create",
"params": [{
"name": "volume1/test"
}]
}
# Create the dataset using the Dataset Creation Message
ws.send(json.dumps(create_dataset_message))
# Receive response from the server
response = ws.recv()
print("Response from server:", response)
main.py →
import functions
# Get the server address entered by the user
server_address_userinput = input("Type the server address: ")
# Continue prompting the user until a valid IP address is provided
while functions.validate_ip_address(server_address_userinput) is not True:
print("Invalid IP address format or out-of-range values. Please try again.")
server_address_userinput = input("Type the server address: ")
# WebSocket endpoint using the address provided by the user
websocket_url = f"wss://{server_address_userinput}/websocket"
# Call the connect and authenticate function with username and password
functions.connect_and_authenticate(websocket_url)
# Call the Create Dataset Function
functions.create_dataset(websocket_url)
# Call the close connection function
functions.close_connection(websocket_url)
The return from server (in the last function “create_dataset”) is: {“msg”: “failed”, “version”: “1”}
Could someone help to understand what is wrong in the code?
Thank you very much for you attention.
I’m not sure how I’m supposed to use those codes in order to achieve this. Could you point me here I can get sufficient info in order to be able to understand this?
Well, I stopped using the websocket and startet to use the Restful 2.0 API using the request library from python. I got the creation of dataset to work as expected:
def dataset_create_request(api_url, api_key, directory_structure):
# Separation of each key and value from the directory dictionary.
for key, value in directory_structure.items():
current_path = key
# Endpoint for dataset creation
dataset_endpoint = "/pool/dataset"
# Headers with API key
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json"
}
# Payload to create the dataset
dataset_payload = {
"name": f"{current_path}"
}
# Create the dataset
dataset_response = requests.post(api_url + dataset_endpoint, json=dataset_payload, headers=headers,
verify=False)
if dataset_response.status_code == 200:
print(f"Dataset '{current_path}' created successfully!")
else:
print(f"Error creating '{current_path}' dataset:", dataset_response.text)
if isinstance(value, dict):
dataset_create_request(api_url, api_key, value)
elif isinstance(value, list):
for item in value:
current_item_path = item
# Endpoint for dataset creation
dataset_endpoint = "/pool/dataset"
# Headers with API key
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json"
}
# Payload to create the dataset
dataset_payload = {
"name": f"{current_item_path}"
}
# Create the dataset
dataset_response = requests.post(api_url + dataset_endpoint, json=dataset_payload, headers=headers,
verify=False)
if dataset_response.status_code == 200:
print(f"Dataset '{current_item_path}' created successfully!")
else:
print(f"Error creating '{current_item_path}' dataset:", dataset_response.text)
Now I want to perform the creation of ACLs for those datasets. Could you please point to me wich endpoint to use and the structure of the properties in this endpoint in order to create special acls? I mean, I want to use the detailed version of the acls.
You can cd into the src/middlewared/middlewared/client directory and run the command python3 setup.py install (though maybe consider using python virtual environments if this is for automation).
The client supports context manager protocol
from middlewared.client import Client
with Client('ws://192.168.0.225/websocket') as c:
# Authenticate with username / password. Using API key is probably better for automation
c.call('auth.login', <username>, <password>)
# Example of getting stat output of path
st = c.call('filesystem.stat', '/mnt/tank')
# example of getting the ACL on a path
acl = c.call('filesystem.getacl', '/mnt/tank')
# example of doing a recursive chown on a path
# job=True means that we block until it completes
chown_job = c.call(
'filesystem.chown',
{'path': '/mnt/tank/SHARE', 'uid': 1000, 'options': {'recursive': True}},
job=True
)
Once the context manager exits, the websocket session is cleaned up
If you navigate to http://<ip of server>/api/docs you will see full details of API endpoints. auth.login_with_api_key.
If you look at documentation for api_key.create you can see how to specify an allowlist to specify what methods and resources the API key is allowed to call.
When context manager exits, the connection is closed / logged out.
My friend, could you please confirm where I can find the filesystem.setacl endpoint documentation? I am not able to find the schema and the “atributes” to set the permissions…
I am not sure if I can simply repete those configurations that I’ve seen using the get “/pool/dataset/id/{id}”:
See accepts decorator for hints. If you are unfamiliar with what these things are though I think writing automation wrappers for them is probably not a good place to start. I’ve seen a lot of automated stuff doing the wrong thing.
To clarify a bit more. ACL operations are in filesystem namespace and not pool.dataset because they are only relevant when a dataset is mounted (they are path-based). You cannot change the permissions on an unmounted dataset. If you want to change permissions on a mounted dataset you need to look at its mountpoint.
