Housekeeping… Remember to stop by to say hi to our sponsors! Find one of our organizers and take a selfie with them Isaac, Adam, Prashant, Gunjan, or Dan Don’t forget to tag it #SPSDC or #CLOUDSATDC WiFi Information SSID: MSFTGuest Wi-Fi Code: msevent475kp

Thanks to all our Sponsors!!!

Join us at #SharePint! Why? To network with fellow Microsoft professionals What? SharePint!!! When? 5:45 PM Where? Lets take over the Crafthouse at Reston Town Center!

Manipulating Share Point Lists using curl Presented by don@hautsch.com

Agenda Use cases Authenticating Upload/Download files SP List Metadata CRUD - Create, read, update and delete Scripts

Use cases You want to download or upload files between SharePoint and Linux easily You want to manipulate SharePoint lists from Linux easily Read email attachments on Linux using java classes microsoft.exchange.webservices.data.* From outlook.office365.com Upload the attachments to SharePoint

Authenticating On premises you can use the --ntlm with curl to easily upload or download a file. Create an entry in your $HOME/.netrc file machine sharepoint login ACME\scott password tiger To upload or download SP_SHRD_DOCS=http://sharepoint/sites/eng/Shared%20Documents curl -n --ntlm --upload-file $HOME/my_file.txt $SP_SHRD_DOCS/myfile.txt curl -n --ntlm -o $HOME/my_file.txt $SP_SHRD_DOCS/myfile.txt

Authenticating to sharepoint.com Authenticating at sharepoint.com is a bit harder than on premises Authentication via https://en.wikipedia.org/wiki/Federated_identity This script https://github.com/dhautsch/etc/blob/master/scripts/xfer_sp_file will handle file upload or download for local or cloud SharePoint SP_SHRD_DOCS=https://acme.sharepoint.com/sites/etl/Shared%20Documents xfer_sp_file /etc/hosts $SP_SHRD_DOCS/hosts.txt xfer_sp_file $SP_SHRD_DOCS/hosts.txt /tmp/hosts.txt

Authenticating to sharepoint.com continued Most likely you will have to deal with a HTTPS proxy Set the following environment variable for curl export HTTPS_PROXY=https://proxy.acme.com:8080 Your $HOME/.netrc will have to have the site name from sharepoint.com machine acme.sharepoint.com login ACME\scott password tiger

Authenticating to sharepoint.com continued Create a saml request file saml.xml to get a security token <s:Envelope xmlns:s='http://www.w3.org/2003/05/soap-envelope' xmlns:a='http://www.w3.org/2005/08/addressing' xmlns:u='http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd'> <s:Header> <a:Action s:mustUnderstand='1'>http://schemas.xmlsoap.org/ws/2005/02/trust/RST/Issue</a:Action> <a:ReplyTo><a:Address>http://www.w3.org/2005/08/addressing/anonymous</a:Address></a:ReplyTo> <a:To s:mustUnderstand='1'>https://login.microsoftonline.com/extSTS.srf</a:To> <o:Security s:mustUnderstand='1' xmlns:o='http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd'> <o:UsernameToken><o:Username>scott@acme.com</o:Username><o:Password>tiger</o:Password></o:UsernameToken> </o:Security> </s:Header><s:Body> <t:RequestSecurityToken xmlns:t='http://schemas.xmlsoap.org/ws/2005/02/trust'> <wsp:AppliesTo xmlns:wsp='http://schemas.xmlsoap.org/ws/2004/09/policy'> <a:EndpointReference><a:Address>acme.sharepoint.com</a:Address></a:EndpointReference> </wsp:AppliesTo> <t:KeyType>http://schemas.xmlsoap.org/ws/2005/05/identity/NoProofKey</t:KeyType> <t:RequestType>http://schemas.xmlsoap.org/ws/2005/02/trust/Issue</t:RequestType> <t:TokenType>urn:oasis:names:tc:SAML:1.0:assertion</t:TokenType> </t:RequestSecurityToken> </s:Body></s:Envelope>

Authenticating to sharepoint.com continued Make request to get security token curl --connect-timeout 10 --max-time 120 -n --proxy-anyauth -o curl_reponse.xml \ --data @saml.xml --ciphers AES256-SHA \ -H 'accept:application/atom+xml;charset=utf-8' \ https://login.microsoftonline.com/extSTS.srf Extract token from file and deswizzle the string, convert “&” to “&” SAML_TOKEN=$( perl -lane 'print $1 if m!<wsse:BinarySecurityToken[^>]*>(.*)</wsse:BinarySecurityToken!' curl_reponse.xml | \ perl -p -e 's/&/&/sg;s/</</sg;s/>/>/sg;s/"/"/sg;' )

Authenticating to sharepoint.com continued Make a form sign in request to get cookies, these cookies last a while echo $SAML_TOKEN > form.dat curl --connect-timeout 10 --max-time 120 -n --proxy-anyauth -o /dev/null \ -c cookies.txt \ --data @form.dat \ --ciphers AES256-SHA \ -H "HOST:acme.sharepoint.com" \ -L "https://acme.sharepoint.com/_forms/default.aspx?wa=wsignin1.0" COOKIE1=$(perl -lane 'print "$1=$2" if m!(rtFa)\s+(\S+)!' cookies.txt) COOKIE2=$(perl -lane 'print "$1=$2" if m!(FedAuth)\s+(\S+)!' cookies.txt) Now it gets easy again

Updating a SharePoint list needs a digest Request an authorization digest from SharePoint which will last a while but expire before the cookies, for example, 1 hour versus 8 hours. CLOUD_SP=http://acme.sharepoint.com/sites/eng curl --connect-timeout 10 --max-time 120 -n --proxy-anyauth \ --data "''" \ --ciphers AES256-SHA \ -H "Cookie:$COOKIE1;$COOKIE2" \ -H 'accept:application/atom+xml;charset=utf-8' \ -o digest.xml \ "$CLOUD_SP/sites/eng/_api/contextinfo" DIGEST=$(perl -lane 'print "Authorization:Bearer $1" if m!<d:FormDigestValue[^>]*>(.*)</d:FormDigestValue!' digest.xml) DIGEST_TIMEOUT=$(perl -lane 'print $1 if m!<d:FormDigestTimeoutSeconds[^>]*>(.*)</d:FormDigestTimeoutSeconds!' digest.xml)

Upload/Download file sharepoint.com curl --connect-timeout 10 --max-time 120 -n --proxy-anyauth \ -H "Cookie:$COOKIE1;$COOKIE2" \ -H "$DIGEST" \ --ciphers AES256-SHA \ --data-binary @my_file.txt \ -o curl_response.xml \ "$CLOUD_SP/_api/web/getfolderbyserverrelativeurl('Shared%20Documents')/Files/Add(url='my_file.txt',overwrite=true)" Download -o my_file.txt \ "$CLOUD_SP/Shared%20Documents/my_file.txt"

SP List Metadata - Get metadata, write ListItemEntityTypeFullName to dot file curl -s --max-time 30 -n --proxy-anyauth \ --dump-header curl_header.txt \ --ciphers AES256-SHA \ -H 'accept:application/atom+xml;charset=utf-8' \ -H "Cookie:$COOKIE1;$COOKIE2" \ -H "$DIGEST" \ -o curl_response.xml \ "$CLOUD_SP/_api/lists/getByTitle('Bogus')" for s in Created EntityTypeName Id ItemCount \ LastItemDeletedDate LastItemModifiedDate \ LastItemUserModifiedDate ListItemEntityTypeFullName Title do perl -lane 'print "$1=$2" if m!<d:('$s')[^>]*>([^<]+)!' curl_response.xml done | tee dot_file.txt

CRUD - Create list item, write Id to dot file source dot_file.txt curl -s --max-time 30 -n --proxy-anyauth \ --dump-header curl_header.txt \ --ciphers AES256-SHA \ -H 'accept:application/atom+xml;charset=utf-8' \ -H "Cookie:$COOKIE1;$COOKIE2" \ -H "$DIGEST" \ -o curl_response.xml \ -X POST \ --data "{ '__metadata': { 'type': '$ListItemEntityTypeFullName' }, 'Title': 'New-18999' }" \ -H 'content-type:application/json;odata=verbose' \ -H 'IF-MATCH:*' \ "$CLOUD_SP/_api/lists/getByTitle('Bogus')/items" for s in Created Id Modified Title do perl -lane 'print "$1=$2" if m!<d:('$s')[^>]*>([^<]+)!' curl_response.xml done | tee dot_file.txt

CRUD - Update list item, look for 204 in header source dot_file.txt curl -s --max-time 30 -n --proxy-anyauth \ --dump-header curl_header.txt \ --ciphers AES256-SHA \ -H 'accept:application/atom+xml;charset=utf-8' \ -H "Cookie:$COOKIE1;$COOKIE2" \ -H "$DIGEST" \ -o curl_response.xml \ -X MERGE \ --data "{ '__metadata': { 'type': '$ListItemEntityTypeFullName' }, 'Title': 'Updated-18999' }" \ -H 'content-type:application/json;odata=verbose' \ -H 'IF-MATCH:*' \ "$CLOUD_SP/_api/lists/getByTitle('Bogus')/items($Id)" egrep '^HTTP' curl_header.txt

CRUD - Read list item, note we are using $filter rather than /items($Id) curl -s --max-time 30 -n --proxy-anyauth \ --dump-header curl_header.txt \ --ciphers AES256-SHA \ -H 'accept:application/atom+xml;charset=utf-8' \ -H "Cookie:$COOKIE1;$COOKIE2" \ -H "$DIGEST" \ -o curl_response.xml \ "$CLOUD_SP/_api/lists/getByTitle('Bogus')/items?%24filter=ID%20eq%20$Id" for s in Created Id Modified Title do perl -lane 'print "$1=$2" if m!<d:('$s')[^>]*>([^<]+)!' curl_response.xml done | tee dot_file.txt

CRUD - Delete list item, look for 200 in header curl -s --max-time 30 -n --proxy-anyauth \ --dump-header curl_header.txt \ --ciphers AES256-SHA \ -H 'accept:application/atom+xml;charset=utf-8' \ -H "Cookie:$COOKIE1;$COOKIE2" \ -H "$DIGEST" \ -o curl_response.xml \ -X DELETE \ -H 'IF-MATCH:*' \ "$CLOUD_SP/_api/lists/getByTitle('Bogus')/items($Id)" egrep '^HTTP' curl_header.txt

Scripts The CRUD examples are from https://github.com/dhautsch/etc/blob/master/scripts/sp_curl_examples For a general perl script that works on Linux and Sun https://github.com/dhautsch/etc/blob/master/perl/get_sp_list_items.pl For a perl script that updates a file from list when list is newer https://github.com/dhautsch/etc/blob/master/perl/update_file_from_sp.pl

Scripts - get_sp_list_items.pl examples get_sp_list_items.pl -create -data "{ '__metadata': { 'type': 'SP.Data.BogusListItem' }, 'Title': 'New_bogus-$$' }" \ http://sharepoint/sites/etl Bogus get_sp_list_items.pl -update -id $ID -data "{ '__metadata': { 'type': 'SP.Data.BogusListItem' }, 'Title': 'Updated-$$' }" \ echo "{ '__metadata': { 'type': 'SP.Data.BogusListItem' }, 'Title': 'New_bogus-$$' }" | get_sp_list_items.pl -create -data @- \ echo "{ '__metadata': { 'type': 'SP.Data.BogusListItem' }, 'Title': 'Updated-$$' }" | get_sp_list_items.pl -update -id $ID -data @- \ get_sp_list_items.pl -create -data "@create_data.txt" http://sharepoint/sites/etl Bogus get_sp_list_items.pl -update -id $ID -data "@update_data.txt" http://sharepoint/sites/etl Bogus get_sp_list_items.pl -delete -id $ID http://sharepoint/sites/etl Bogus get_sp_list_items.pl -meta http://sharepoint/sites/etl Bogus get_sp_list_items.pl -digest http://sharepoint/sites/etl get_sp_list_items.pl http://sharepoint/sites/etl Bogus get_sp_list_items.pl -query '$top=5' http://sharepoint/sites/etl Bogus

Scripts - get_sp_list_items.pl called from python

Scripts - get_sp_list_items.pl usage