Connecting Google Cloud Storage with Lightcast Data Shares

Lightcast offers the world’s most comprehensive data and insights on labor market trends, job postings, workforce profiles, compensation, career pathways, skills projections, and demographics. Following the launch of Lightcast Data Shares, slicing, dicing, and analyzing large datasets is more seamless than ever before. 

What are Lightcast Data Shares?

Through Lightcast Data Shares, Lightcast data is now available directly on Google Cloud Storage (GCS) and other top file storage platforms, data warehouses, and marketplaces.  

Data sharing through Lightcast is flexible and secure. Lightcast makes the process of sharing data simple for clients and they also have the option to slice, edit, or prepare data before integrating it into internal or external solutions. Because data is shared directly with each client’s existing warehouse or cloud storage as platform-native sharing, a data feed, or both, information shared through Lightcast Data Shares is always up-to-date. 

Accepting data shares from Lightcast requires almost no internal technical resources from customers, resulting in lower costs and increased efficiencies. Data shares streamline the process of using Lightcast data for ingestion into large language models (LLMs), AI solutions, and predictive analytics models. 

Depending on each client’s specific business needs, Lightcast Data Shares can also be customized. Each data share integration can be fully operational within two business days.

How Data Shares and Google Cloud Storage Work Together

Google Cloud Storage (GCS), an object storage service offered by Google Cloud Platform, is one of several delivery destinations supported by Lightcast Data Shares. Lightcast has very flexible support for Google Cloud Storage Buckets, including control over:

• Bucket region

• Whether the bucket is created and managed by Bobsled, or is an external bucket managed by you or a partner

• Access control. The permissions granted allow each Google Principal to perform read and copy operations on the Bobsled-managed destination bucket

What if I Don’t Use Google Cloud Storage? 

Lightcast supports several data share destinations beyond Google Cloud Storage, including Snowflake, Databricks, Google BigQuery, Google Cloud Storage, Microsoft Azure Blob Storage, SFTP, and Amazon S3.

How to Connect Google Cloud Storage with Lightcast Data Shares

Before configuring a destination, a share must be created. Lightcast requires Google Principal(s) to be provided to grant access to the data in the Lightcast-managed destination.

Before consuming a data transfer, a data transfer must be sent to the destination and access must be configured in Lightcast for the identity that is consuming the data.

Consuming a data transfer

From the Shares list page, click on the share that you would like to access.

Once a data transfer in the share has been completed, select the button Access Data.

Option 1: Accessing Data via Web Console

You can easily access the data using the web console link to view and download the data. To do so, please be sure to log in to the [insert cloud’s console name] with the account that has been configured to access the Bobsled share.

Select the Web console tab in the access dialog

Click the link icon to view the data in the GCP Web Console.

Option 2: Accessing Data via Command line

Using the Google Cloud Storage command-line tools (Gsutil or Gcloud), you can list, copy, and sync the contents of the data transfer in Google Cloud Storage. To use the following commands, you will need to copy the Cloud Storage URI located in the access data dialog as pictured above.

Login to the GCP Command-line

1. Run the login command

The login command for Google Cloud Storage is gcloud auth login

If you would like to access the data as a service account, run the command:

gcloud auth activate-service-account [service-account-email] --key-file=[path-to-private-key-file]

Access via Gcloud command-line tool

1. List the contents

To list the data in the bucket, you will use the command gcloud ls

gcloud storage ls -r <storage-bucket-URI>

• Parameters used: -r (recursive) lists all objects in a bucket

• Optional parameters to use with the list command: -l (info) additional information about the bucket (object size, creation time, etc)

2. Copy the contents to your bucket

To copy the data in the bucket, you will use the command gcloud cp

gcloud storage cp -r <storage-bucket-URI> <your bucket/path>

• Parameters used: -r (recursive) copies entire directory tree

• Optional parameters to use with the copy command: -n (no-clobber) prevent overwriting the content of existing files at the destination.

3. Sync the contents

Please use gsutil to sync the contents of the data transfer to your bucket.

Access via Gsutil command-line tool

1. List the contents

To list the data in the bucket, you will use the command gsutil ls

gsutil ls -r <storage-bucket-URI>

• Parameters used: -r (recursive) lists all objects in a bucket

• Optional parameters to use with the list command: -l (info) additional information about the bucket (object size, creation time, etc)

2. Copy the contents to your own bucket

To copy the data in the bucket, you will use the command gsutil cp

gsutil cp -r <storage-bucket-URI> <your bucket/path>

• Parameters used: -r (recursive) copies the entire directory tree

• Optional parameters to use with the copy command: -n (no-clobber) prevent overwriting the content of existing files at the destination

3. Sync the contents

Use sync if you would like to copy only files that are new or updated. Learn more about the command rsync

gsutil rsync <storage-bucket-URI> <your bucket/path>

• Parameters used: -r (recursive) copies the entire directory tree

Learn more about how Lightcast Data Shares work or get in touch with our team to discuss your specific business requirements and connect Google Cloud Storage with Lightcast Data Shares.