Connecting Amazon S3 with Lightcast Data Shares

Lightcast’s labor market trends, job posting, workforce profiles, compensation, career pathways, skills projections, demographic insights, and related insights offer the most comprehensive labor market data in the world. Quickly slicing, dicing, and analyzing large datasets is now easier than ever thanks to the launch of Lightcast data shares. 

What are Lightcast Data Shares?

Lightcast data is now available through direct data shares on Amazon S3 and other leading file storage platforms, data warehouses, and marketplaces.  

Data sharing through Lightcast is seamless, flexible and secure.  Data is shared directly with existing file storage platforms or data warehouses, without requiring technical lifting from clients’ internal teams.

Delivering data through data sharing is effortless. Customers slice, modify, or prepare data before integrating it into their internal or external solutions. Shared data is always current and secure, because data is shared directly with each customer’s existing cloud storage or warehouse as platform-native sharing, a data feed, or both. 

Accepting data shares from Lightcast requires few technical resources, which leads to decreased costs and faster processes. Data shares make the process of using Lightcast data for ingestion into large language models (LLMs), AI solutions, and predictive analytics models easier  for customers.

Lightcast data shares can also be tailored based on each customer’s specific needs and each data share integration can be fully operational in less than two business days.

How Data Shares and Amazon S3 Work Together

Amazon S3, an object storage service offered by Amazon Web Services (AWS), is one of many delivery destinations that Lightcast supports. Lightcast has very flexible support for S3 Buckets, including control over the following

• Bucket region

• Whether the bucket is created and managed by Lightcast, or is an external bucket managed by you or a partner

• Access control

What if I Don’t Use Amazon S3? 

Lightcast supports several data share destinations beyond Amazon S3, including Snowflake, Databricks Google BigQuery, Google Cloud Storage, Microsoft Azure Blob Storage, and SFTP.

How to Connect Amazon S3 with Lightcast Data Shares

A convenient way to deliver data to S3 buckets is to use a Lightcast managed bucket. These are buckets, created and managed by Lightcast, in an AWS account also created and managed by Lightcast. Once created, simply grant Amazon Resource Name (ARNs) access to the data.

For greater flexibility and control, Lightcast can also deliver files to buckets on third-party AWS accounts.

Consuming a Data Share in Amazon S3

Multiple options are available to access a data transfer via a data consumer's command-line interface:

Option 1: Accessing Data via Web Console

Consumers can easily access the data using the web console link to view and download the data.

1. Select the Web console tab in the access dialog

2. Copy the AWS Web Console link icon and send it to your customer

3. The customer will be prompted to log in to their AWS console if they are not already

Option 2: Accessing Data via Command line

Using the AWS command-line tool, consumers can list, copy, and sync the contents of the data transfer in Amazon S3. Lightcast provides out of the box list, copy, and sync commands tailored to the data consumer's bucket.

1. Log In to the CLI: 

Run the command “aws configure”  and you will be prompted on the CLI to enter your AWS Access Key ID, AWS Secret Access Key, default region, and output format.

• For the default region, be sure that it is set the same region as in the Lightcast share.

• Set the output format to json.

2. List the contents

Additional parameters to use with the list command:

• --recursive (boolean) Command is performed on all files or objects under the specified directory or prefix

• --human-readable (boolean) Display file sizes in human readable format

• --summarize (boolean) Displays summary information such as number of objects, total size, etc.

3. Copy the contents to your own bucket

To copy data into your own S3 bucket, the bucket must be in the same region as the Lightcast-managed bucket. If the region is not the same, you will receive an ‘Access Denied’ error.

4. Sync the contents

Use sync if you would like to copy only files that are new or updated.

Discover more about Lightcast data shares or contact our team to discuss your specific needs and how to connect Amazon S3 with Lightcast data shares.