Using AWS Lambda functions with the Salesforce Bulk API


One common task when integrating Salesforce with customers system is to import data, either as a one time task or regularly.

This can be done in several ways depending on the inhouse technical level and the simplest way might be to use the Import Wizard or the Data Loader. If you want to do it regularly in a batch fashion and are fortunate enough to have AWS infrastructure available using Lambda functions is an alternative.

Recently I did this as a prototype and I will share my findings here.

I will not go into details about AWS and Lambda, I used this tutorial to get started with Lambda functions but most of it didn’t concern the Salesforce parts but rather AWS specifics like IAM.

I found this Heroku project for using the bulk api.

The full python code looks like this:

from __future__ import print_function
from base64 import b64decode
import boto3
import uuid
import csv
import os
from salesforce_bulk import SalesforceBulk, CsvDictsAdapter
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

s3_client = boto3.client('s3')
username = os.environ['SF_USERNAME']
encrypted_password = os.environ['SF_PASSWORD']
encrypted_security_token = os.environ['SF_SECURITYTOKEN']
password = boto3.client('kms').decrypt(CiphertextBlob=b64decode(encrypted_password))['Plaintext'].decode('ascii')
security_token = boto3.client('kms').decrypt(CiphertextBlob=b64decode(encrypted_security_token))['Plaintext'].decode('ascii')
mapping_file_bucket = os.environ['MAPPING_FILE_BUCKET']
mapping_file_key = os.environ['MAPPING_FILE_KEY']

def bulk_upload(csv_path, mapping_file_path):
    with open(csv_path, mode='r') as infile:
        logger.info('Trying to login to SalesforceBulk')
        job = None
        try:
            bulk = SalesforceBulk(username=username, password=password, security_token=security_token)
            job = bulk.create_insert_job("Account", contentType='CSV')

            # Mapping file
            mapping_file = open(mapping_file_path, 'rb')
            bulk.post_mapping_file(job, mapping_file.read())

            accounts = csv.DictReader(infile)
            csv_iter = CsvDictsAdapter(iter(accounts))
            batch = bulk.post_batch(job, csv_iter)
            bulk.wait_for_batch(job, batch)
            bulk.close_job(job)
            logger.info('Done. Accounts uploaded.')
        except Exception as e:
            if job:
                bulk.abort_job(job)
            raise e
        return 'OK'

def handler(event, context):
    for record in event['Records']:
        # Incoming CSV file
        bucket = record['s3']['bucket']['name']
        key = record['s3']['object']['key']
        download_path = '/tmp/{}{}'.format(uuid.uuid4(), key)
        s3_client.download_file(bucket, key, download_path)

        # Mapping file
        mapping_file_path = '/tmp/{}{}'.format(uuid.uuid4(), mapping_file_key)
        s3_client.download_file(mapping_file_bucket, mapping_file_key, mapping_file_path)

        return bulk_upload(download_path, mapping_file_path)

Make sure to add the following environment variables in Lambda before executing

SF_USERNAME - your SF username
SF_PASSWORD - your SF password (encrypted)
SF_SECURITYTOKEN - your SF security token (encrypted)
MAPPING_FILE_BUCKET - the bucket in where to find the mapping file
MAPPING_FILE_KEY - the path to the mapping file in that bucket

I also added a method (In my own clone of the project here) to be able to provide the mapping file as part of the payload, I’ll make sure to create a pull request for this later.

The nice thing with using the Bulk API is that you get the monitoring directly in Salesforce, just go to to see the status of your job(s).

I haven’t added the listen to S3-trigger yet but it’s the next part of the tutorial so shouldn’t be a problem.

Cheers,
Johan

Trailhead is awesome and gamification totally works!

About this time last year I decided to pursue a career within Salesforce, I was a bit tired of my current job and wanted a change. It was either a backend engineer role at iZettle or becoming a Salesforce Consultant. The consultant role was not new to me since that’s how I started my career. After signing the contract I decided to look at Trailhead since I heard a lot of good things about it.

I took my first badge, Salesforce Platform Basics at 2016-10-04 20:32 UTC and it was quite easy. My goal was to take a bunch before I started working on the 2:nd of January 2017. That didn’t happen but I started doing them from my first day in the office.

Having worked with Salesforce since 5/11/2012 I thought I knew most of the platform but that was far from the truth. Platform Cache, Hierarchical Custom settings, Shield, Communities, Live Agent, etc. I haven’t worked with much of it but today I still know how the features works and if a customer asks me about it I can at give a brief explanation about most things.

Back to the gamification part, at EINS we set a goal for 2017 to have at least 98% of all badges between us in the team. Using partners.salesforce.com for calculating this value is really hard so I set out to build a dashboard for the team so that we could see our score.

After some iterations it looks ok, Lightning Design System makes everything looks great.
Trailhead Tracker

It looks awesome on your phone too

The dashboard has really helped, mostly because people in the team now sees who took a badge over the weekend and when someone passes their total number of badges. This has encouraged everyone to go that extra mile and take that extra badge.

Healthy competition is always good and when you learn things that helps you make a better job while at it it’s definitely a win-win!

Feel free to check out the dashboard at http://trailhead.eins.se/, also if you click a persons name you can see which badges he/she is missing. This makes it easy when you’re looking for a quick badge to take on the subway or the bus.
Trailhead Tracker User Page

Another addition was the #architectjourney pyramid at the bottom, since we’re scraping certifications too
Trailhead Tracker Architect Journy

The last thing we added to the dashboard was Slack notifications when someone completes a badge or gets a Salesforce Certification, of course the first version of this spammed all over our #trailhead channel but that bug is long gone now.
Trailhead Tracker Slack

Exporting the data in CSV and importing it into Wave lets you gain some insights into when people take badges (and when they work)
Badges per Month

So in summary, Trailhead gamification totally works but you need a dashboard.

Cheers,
Johan

PS. My aim is to clean up the code and put it on GitHub when I have the time DS.

Using Jenkins and Git for Metadata backups and running Test Cases

One thing that makes Salesforce great is the possibility to use it and customize it quite far without having to invite developers. I can see the beauty of this since developers are expensive and slow, before you think about writing a comment on that last statement keep in mind I’m a developer myself. One thing that code developers (as opposed to click developers) bring to the table is source control. This is mostly because anyone who has ever written a piece of code bigger than “hello world” knows that it’s super hard to get code right the first time, or the second time, and so on.

Coming from the coding part of the world developers love source control, code reviews, continuous integration and so on. For a Salesforce using company without an in house development team this is usually not something they ever think about.

Before I continue my rambling, my point is that Source Control is something anyone can benefit from, especially since the Metadata API in Salesforce makes it very easy to retrieve everything via the API.

I have created a small GitHub project here.
If you have a Jenkins instance to spare you should be up and running within the hour taking a backup of your Metadata and also running your test cases (if you have any Apex code) to make sure a declarative change didn’t break your test cases, which you would find out about the next time you’re trying to deploy something in your very tight 1 hour service window (yes that will happen).

In short, clone the GitHub project and follow the README. If you don’t have a Jenkins instance you can easily find one here. I had an old VMWare server where I just deployed their VMWare image.

Having regular Metadata backups is great, especially if you have more than 1 Salesforce Administrator. It might not help you in a proactive way but you’ll get a full audit trail on when a Profile was changed or when a column in a List View disappeared.

If you run into any problems, just let me know and I’ll be happy to assist you, no one should run their Salesforce instance without regular backups.

I used this GitHub project as a starting point, big thanks to @JitendraZaa for doing most of the heavy lifting.

Cheers,
Johan

Time for a new start

So I haven’t posted anything for 4 years and I think it’s time for a fresh start. Nowadays I work as a Salesforce Consultant and I will try to share any knowledge I get about the force.com platform and hopefully it will be helpful. As any other developer I learn a lot of things by reading blogs and Stackoverflow so I’m hoping to give back in some way to the community.

Cheers,
Johan

Mass deleting picklist values in Salesforce with AJAX javascript hack

Today I was gonna do a small change in our Salesforce instance, namely replacing to 1100+ picklists with brand new values. Last time I did this I used Eclipse and just saved the file (Lead.object for example).

But I realized that only new values are added and old ones are not removed and as usual there are number of requests on success.salesforce.com for this feature but it is not implemented yet.

Anyways after Googling I found this article:
http://frombelvideres4thfloor.blogspot.se/2011/05/javascript-hack-to-mass-delete-custom.html
which basically solves the problem by opening new windows.

This seemed to put a lot of stress on my poor MacBook so I edited it to use synchronous AJAX requests instead.


/* Gather all of the a elements on the page. */

var links = document.getElementsByTagName("a");

/* Pick out the Del links. */

var delLinks = new Array();
for (var i = 0; i < links.length; i++) {
  var link = links[i];

  if (link.innerHTML == "Del") {
    /*alert("Del link found!");*/
    /*alert(link.attributes['href'].value);*/
    delLinks[delLinks.length] = link;
  }
}

/* AJAX request each Del link to delete the associated
   picklist value.

   This code can be augmented as desired
   to only delete certain values.

   However, for custom picklists it's probably
   easier to just delete all of the values
   and then re-add the desired values. */

for (var i = 0; i < delLinks.length; i++) {
  var delLink = delLinks[i];
  // Synchronous AJAX style
  xmlhttp = new XMLHttpRequest();
  xmlhttp.open("GET",delLink.attributes['href'].value,false);
  xmlhttp.send();
}

The last 3 lines are what I’ve changed and it will call all delete links on a picklist configuration page. To get it running just create a new bookmark in your browser.

javascript:(function()%7Bvar s = document.createElement("script"); s.src = "https://superfredag.com/massdelete.js"; void(document.body.appendChild(s));%7D)()

And replace the url with where you put your javascript file, or use mine if you want to.

Then just go to a picklist edit page and click it, it will take a while to execute but you can see that things are happening by opening Developer Tools (I’m using Chromium) and go to the Network tab.

Use it with caution though, it worked for me but you never know when Salesforce changes the layout. Test in your staging environment first 😉

This helped me remove 1100+ picklist values with very little effort.

Big thanks to  for writing the original post.

Why you should have a theme song (or why it is important to be happy at work)

Long time no see, a lot has happened and at the moment I work in Stockholm and live in a basement which is perfect.

I have thought about this many times before and this morning it struck me once again: You should have a theme song in your head when you wake up and on your way to work.

Say what? Is probably the sane reaction to my last sentence but let me explain.

I love what I do and I really like my job more or less every day but somedays it just feels like I don’t wanna get out of the bed, walk 15 minutes, take a train, take a bus and finally reach my destination for some free breakfast. On other days I wake up feeling invincible, like I could take on the world with my bare hands and on my way to work on those days I get in the mood listening to music.

I have a lot of friends who only show up at work to get their paycheck which is really sad, after a couple of years they have a house (mortgage), kids, a car and a wife and they still go to the same job to get the money to support all of it. This is of course very good and I don’t think they’re unhappy since they value their life outside of the workplace more than the actual work.

But for me my job has always been a big part of my life and if I don’t feel like going to work for a long period of time I really have to find a new job. If I’m unhappy where I spend at least 40 hours a week I will be unhappy the rest of the time and this will affect everyone around me.

So I’m so happy when I wake up feeling like: “cue Rocky theme” (Eye of the tiger) and I get ready for work listening to The Transformers soundtrack or equivalent.

And if you haven’t woken to you personal theme song (yes you can choose your own) in a long time then you should at least considering changing something in your life because a life with a theme song is a good life.

Using SHA-1 to generate passwords

This is probably the worst thing about the Internet, remembering passwords.

Until about a year ago I had the same password for almost every service, sometimes I spiced it up by adding a character in the start representing the service (F for facebook, G for Gmail and so on).

A colleague had a different approach, he used simple “passwords” which he then computed SHA-1 on. He had developed a simple java script that performed the SHA-1 conversion.

I have changed his original script a bit to be able to generate different length passwords (11, 21 and 26 characters).

You can try it out yourself, just add the following url as a bookmark (works in all browsers, I think):

javascript:(function()%7Bvar%20s%20=%20document.createElement(%22script%22);%20s.src%20=%20%22https://superfredag.com/convert.js%22;%20void(document.body.appendChild(s));%7D)()

To use it, put a “-” in the password box and press the link, this will give you a glass pane with a password box, just write your simple password (Facebook1234 for example) and choose the length, the script will enter the password in the field where you put the “-“.
If you choose Facebook1234 as your simple password the 21 character SHA-1 will be fc1a17377c7ed19872037.

The convert.js script is YUI-compressed, the original one is convert.js.original.
There is also a form based version if you just want the password in cleartext:

https://superfredag.com/convert.html

Of course everything is on GitHub if you want to look at it and put it up for yourself. If you don’t have any web hosted you could just put the scripts in the Public part of your Dropbox.

Polygon with holes in aggdraw

Drawing polygons with holes have haunted me for a while and recently I had to do it in Python with PIL.

Googling the subject gives you a hint.

This code:

import os
import Image
import aggdraw

draw = aggdraw.Draw('RGB', (100, 100), 'white')

path = aggdraw.Path()
path.moveto(10, 10)
path.lineto(10,60,60,60)
path.lineto(60,10)
path.lineto(10,10)
path.moveto(20,20)
path.lineto(40,20)
path.lineto(40,40)
path.lineto(20,40)
path.lineto(20,20)

pen = aggdraw.Brush("black")
draw.path((25, 25), path, pen, None)

img = Image.fromstring('RGB', (100, 100), draw.tostring())
p = os.path.join(os.path.dirname(__file__), 'box.png')
img.save(p)
 Gives the following image:
Polygon with hole created with aggdraw

 

Compiling scipy in 32 bit in a 64 bit environment (el5)

During the last 2 days I’ve been trying to compile an old product in 32 bit mode on a 64 bit Redhat Enterprise Linux 5 environment which should not be that hard.

Python itself is no problem:

TCC="gcc -m32" ./configure

(got information from here)

And this approach works for almost every 3rd party software, except for Scipy.

Scipy contains a lot of FORTRAN code and it wasn’t obvious how to get setup.py to understand that it should both build and link with the -m32 flag.

After a lot of trial and error this is what I used:

F90FLAGS="-m32" F77FLAGS="-m32" \
LDFLAGS="-g -Wall -shared -m32 -fPIC" \
$PYTHON setup.py config_fc --fcompiler=gnu95 install

There is probably other flags that are better but these worked for me, I think the LDFLAGS is what did it since they are used when g77 is linking the FORTRAN code. The flags also worked fine for numpy.

I will try to write more often but I have had a lot to do recently.

Using your smartphone in the US on vacation (T-mobile rocks)

This is something that has been discussed on forums everywhere and there is no real answer.

Just arrived in San Francisco with my swedish iPhone 4, went into a T-mobile store and bought a Monthly4G package for 50$, this gives me unlimited data, talk and text for one month.

Added 10$ and I got free text and landline calls to Sweden.

The only bad thing is that T-mobile 3G network uses a frequency that the iPhone (and many smartphones) cannot use so I’m limited to Edge data transfer.

But still this is very cheap and if you travel with someone you can call them for free while on vacation.