1. Housekeeping… Remember to stop by to say hi to our sponsors!
Find one of our organizers and take a selfie with them
Isaac, Adam, Prashant, Gunjan, or Dan
Don’t forget to tag it #SPSDC or #CLOUDSATDC
WiFi Information
SSID: MSFTGuest
Wi-Fi Code: msevent475kp

2. Thanks to all our Sponsors!!!

3. Join us at #SharePint!
Why? To network with fellow Microsoft professionals
What? SharePint!!!
When? 5:45 PM
Where? Lets take over the Crafthouse at Reston Town Center!

4. Manipulating Share Point Lists using curl
Presented by

5. Agenda Use cases Authenticating Upload/Download files SP List Metadata
CRUD - Create, read, update and delete
Scripts

6. Use cases
You want to download or upload files between SharePoint and Linux easily
You want to manipulate SharePoint lists from Linux easily
Read attachments on Linux using java classes
microsoft.exchange.webservices.data.*
From outlook.office365.com
Upload the attachments to SharePoint

7. Authenticating
On premises you can use the --ntlm with curl to easily upload or download a file.
Create an entry in your $HOME/.netrc file
machine sharepoint login ACME\scott password tiger
To upload or download
SP_SHRD_DOCS=
curl -n --ntlm --upload-file $HOME/my_file.txt $SP_SHRD_DOCS/myfile.txt
curl -n --ntlm -o $HOME/my_file.txt $SP_SHRD_DOCS/myfile.txt

8. Authenticating to sharepoint.com
Authenticating at sharepoint.com is a bit harder than on premises
Authentication via
This script
will handle file upload or download for local or cloud SharePoint
SP_SHRD_DOCS=
xfer_sp_file /etc/hosts $SP_SHRD_DOCS/hosts.txt
xfer_sp_file $SP_SHRD_DOCS/hosts.txt /tmp/hosts.txt

9. Authenticating to sharepoint.com continued
Most likely you will have to deal with a HTTPS proxy
Set the following environment variable for curl
export HTTPS_PROXY=
Your $HOME/.netrc will have to have the site name from sharepoint.com
machine acme.sharepoint.com login ACME\scott password tiger

10. Authenticating to sharepoint.com continued
Create a saml request file saml.xml to get a security token
<s:Envelope xmlns:s='
xmlns:a='
xmlns:u='
<s:Header>
<a:Action s:mustUnderstand='1'>
<a:ReplyTo><a:Address>
<a:To s:mustUnderstand='1'>
<o:Security s:mustUnderstand='1' xmlns:o='
</o:Security>
</s:Header><s:Body>
<t:RequestSecurityToken xmlns:t='
<wsp:AppliesTo xmlns:wsp='
<a:EndpointReference><a:Address>acme.sharepoint.com</a:Address></a:EndpointReference>
</wsp:AppliesTo>
<t:KeyType>
<t:RequestType>
<t:TokenType>urn:oasis:names:tc:SAML:1.0:assertion</t:TokenType>
</t:RequestSecurityToken>
</s:Body></s:Envelope>

11. Authenticating to sharepoint.com continued
Make request to get security token
curl --connect-timeout 10 --max-time 120 -n --proxy-anyauth -o curl_reponse.xml \
--ciphers AES256-SHA \
-H 'accept:application/atom+xml;charset=utf-8' \
Extract token from file and deswizzle the string, convert “&” to “&”
SAML_TOKEN=$(
perl -lane 'print $1 if m!<wsse:BinarySecurityToken[^>]*>(.*)</wsse:BinarySecurityToken!' curl_reponse.xml | \
perl -p -e 's/&/&/sg;s/</</sg;s/>/>/sg;s/"/"/sg;' )

12. Authenticating to sharepoint.com continued
Make a form sign in request to get cookies, these cookies last a while
echo $SAML_TOKEN > form.dat
curl --connect-timeout 10 --max-time 120 -n --proxy-anyauth -o /dev/null \
-c cookies.txt \ \
--ciphers AES256-SHA \
-H "HOST:acme.sharepoint.com" \
-L "
COOKIE1=$(perl -lane 'print "$1=$2" if m!(rtFa)\s+(\S+)!' cookies.txt)
COOKIE2=$(perl -lane 'print "$1=$2" if m!(FedAuth)\s+(\S+)!' cookies.txt)
Now it gets easy again

13. Updating a SharePoint list needs a digest
Request an authorization digest from SharePoint which will last a while but expire before the cookies, for example, 1 hour versus 8 hours.
CLOUD_SP=
curl --connect-timeout 10 --max-time 120 -n --proxy-anyauth \
--data "''" \
--ciphers AES256-SHA \
-H "Cookie:$COOKIE1;$COOKIE2" \
-H 'accept:application/atom+xml;charset=utf-8' \
-o digest.xml \
"$CLOUD_SP/sites/eng/_api/contextinfo"
DIGEST=$(perl -lane 'print "Authorization:Bearer $1" if m!<d:FormDigestValue[^>]*>(.*)</d:FormDigestValue!' digest.xml)
DIGEST_TIMEOUT=$(perl -lane 'print $1 if m!<d:FormDigestTimeoutSeconds[^>]*>(.*)</d:FormDigestTimeoutSeconds!' digest.xml)

14. Upload/Download file sharepoint.com
curl --connect-timeout 10 --max-time 120 -n --proxy-anyauth \
-H "Cookie:$COOKIE1;$COOKIE2" \
-H "$DIGEST" \
--ciphers AES256-SHA \ \
-o curl_response.xml \
"$CLOUD_SP/_api/web/getfolderbyserverrelativeurl('Shared%20Documents')/Files/Add(url='my_file.txt',overwrite=true)"
Download
-o my_file.txt \
"$CLOUD_SP/Shared%20Documents/my_file.txt"

15. SP List Metadata - Get metadata, write ListItemEntityTypeFullName to dot file
curl -s --max-time 30 -n --proxy-anyauth \
--dump-header curl_header.txt \
--ciphers AES256-SHA \
-H 'accept:application/atom+xml;charset=utf-8' \
-H "Cookie:$COOKIE1;$COOKIE2" \
-H "$DIGEST" \
-o curl_response.xml \
"$CLOUD_SP/_api/lists/getByTitle('Bogus')"
for s in Created EntityTypeName Id ItemCount \
LastItemDeletedDate LastItemModifiedDate \
LastItemUserModifiedDate ListItemEntityTypeFullName Title do
perl -lane 'print "$1=$2" if m!<d:('$s')[^>]*>([^<]+)!' curl_response.xml
done | tee dot_file.txt

16. CRUD - Create list item, write Id to dot file
source dot_file.txt
curl -s --max-time 30 -n --proxy-anyauth \
--dump-header curl_header.txt \
--ciphers AES256-SHA \
-H 'accept:application/atom+xml;charset=utf-8' \
-H "Cookie:$COOKIE1;$COOKIE2" \
-H "$DIGEST" \
-o curl_response.xml \
-X POST \
--data "{ '__metadata': { 'type': '$ListItemEntityTypeFullName' }, 'Title': 'New-18999' }" \
-H 'content-type:application/json;odata=verbose' \
-H 'IF-MATCH:*' \
"$CLOUD_SP/_api/lists/getByTitle('Bogus')/items"
for s in Created Id Modified Title do
perl -lane 'print "$1=$2" if m!<d:('$s')[^>]*>([^<]+)!' curl_response.xml
done | tee dot_file.txt

17. CRUD - Update list item, look for 204 in header
source dot_file.txt
curl -s --max-time 30 -n --proxy-anyauth \
--dump-header curl_header.txt \
--ciphers AES256-SHA \
-H 'accept:application/atom+xml;charset=utf-8' \
-H "Cookie:$COOKIE1;$COOKIE2" \
-H "$DIGEST" \
-o curl_response.xml \
-X MERGE \
--data "{ '__metadata': { 'type': '$ListItemEntityTypeFullName' }, 'Title': 'Updated-18999' }" \
-H 'content-type:application/json;odata=verbose' \
-H 'IF-MATCH:*' \
"$CLOUD_SP/_api/lists/getByTitle('Bogus')/items($Id)"
egrep '^HTTP' curl_header.txt

18. CRUD - Read list item, note we are using $filter rather than /items($Id)
curl -s --max-time 30 -n --proxy-anyauth \
--dump-header curl_header.txt \
--ciphers AES256-SHA \
-H 'accept:application/atom+xml;charset=utf-8' \
-H "Cookie:$COOKIE1;$COOKIE2" \
-H "$DIGEST" \
-o curl_response.xml \
"$CLOUD_SP/_api/lists/getByTitle('Bogus')/items?%24filter=ID%20eq%20$Id"
for s in Created Id Modified Title do
perl -lane 'print "$1=$2" if m!<d:('$s')[^>]*>([^<]+)!' curl_response.xml
done | tee dot_file.txt

19. CRUD - Delete list item, look for 200 in header
curl -s --max-time 30 -n --proxy-anyauth \
--dump-header curl_header.txt \
--ciphers AES256-SHA \
-H 'accept:application/atom+xml;charset=utf-8' \
-H "Cookie:$COOKIE1;$COOKIE2" \
-H "$DIGEST" \
-o curl_response.xml \
-X DELETE \
-H 'IF-MATCH:*' \
"$CLOUD_SP/_api/lists/getByTitle('Bogus')/items($Id)"
egrep '^HTTP' curl_header.txt

20. Scripts The CRUD examples are from
For a general perl script that works on Linux and Sun
For a perl script that updates a file from list when list is newer

21. Scripts - get_sp_list_items.pl examples
get_sp_list_items.pl -create -data "{ '__metadata': { 'type': 'SP.Data.BogusListItem' }, 'Title': 'New_bogus-$$' }" \
Bogus
get_sp_list_items.pl -update -id $ID -data "{ '__metadata': { 'type': 'SP.Data.BogusListItem' }, 'Title': 'Updated-$$' }" \
echo "{ '__metadata': { 'type': 'SP.Data.BogusListItem' }, 'Title': 'New_bogus-$$' }" | get_sp_list_items.pl -create \
echo "{ '__metadata': { 'type': 'SP.Data.BogusListItem' }, 'Title': 'Updated-$$' }" | get_sp_list_items.pl -update -id $ID \
get_sp_list_items.pl -create -data Bogus
get_sp_list_items.pl -update -id $ID -data Bogus
get_sp_list_items.pl -delete -id $ID Bogus
get_sp_list_items.pl -meta Bogus
get_sp_list_items.pl -digest
get_sp_list_items.pl Bogus
get_sp_list_items.pl -query '$top=5' Bogus

22. Scripts - get_sp_list_items.pl called from python

23. Scripts - get_sp_list_items.pl usage