Info
Data Source Extracts
Data source extracts are a way to access data stored in Data IQ® more efficiently. Instead of Tableau making a call to Snowflake each time a dashboard is loaded, the data is pre-stored within Tableau Server. This makes loading dashboards much faster since Tableau doesn't have to fetch the data from Snowflake in real time. We are implementing data source extracts into Data IQ® for standard lane throughput data, which will allow interactive throughput reports using a Data Source Extract to open up to 15x faster.
Historical throughput dashboards typically open within 5-7 seconds, depending on data volume and complexity. They are faster than live connection load times, but using multiple data sources, blending, or poorly designed dashboards can cause longer load times.
Our interactive throughput reports, which pull historical data, typically have load times of 30 seconds to 1 minute. However, using data source extracts can significantly speed up the loading process. For example, the Census Historical Trends takes 45 seconds to 1 minute to load, while the Performance Trending Scorecard takes over 1 minute. Using an extract, the Census Historical Trends load within 5 seconds and The Performance Trending Scorecard using an extract loads within 7 seconds.
The extracts will be updated daily at 2 am, covering the last 2 years (730 days) and the previous day. They will not be updated more than once per day. The data refresh time will be managed on a customer-by-customer basis, ensuring Pacific time zone clients receive the correct data refresh time.
The existing standard lane data source connections will be replaced with extracts, causing dashboards connected to these connections to automatically switch to extract data. All fields within the current live connection will be present in the new Data Source Extract connection. Users currently using a standard lane data source will need to switch to either the new 'FULL_HISTORICAL' data source or a fast lane connection.
Fast Lane will continue to operate as usual, updating every 5 minutes, and will utilize Command Center dashboard data source connections. Transfer IQ® data will not be changed and will continue to operate how they currently do.
There will be three types of data source connections: Fast Lane, Historical Extracts, and Standard Lane Full Historical. Fast Lane provides current operations status data, containing 48-72 hours of data depending on the source, with the current status being 5 minutes ago. The historical extracts provide a comprehensive analysis of data trends over the past two years, covering data from 730 days ago to the end of the previous day. The Standard Lane Full Historical provides a comprehensive historical analysis of the improvements made since 2016.
How can I tell if I’m connected to an Extract?
Live connections use a single-cylinder symbol, while extract connections use a double-cylinder symbol.
The diagram below will show what data is retained and the refresh rate of each type of data source.
Transfer IQ® Data | Standard Lane Full Historical Data | Historical Extract Data | Fast Lane Live Data |
Start of data to 35 -45 minutes ago | Start of data to 45 minutes - 1 hour ago | Rolling 2 years (730 days) to end of previous day | 72/48 hours ago to 5 minutes ago |
Normal content load time | Normal content load time | Fast content load time | Normal content load time |
Standard subscription processing time | Standard subscription processing time | Fast subscription processing time | Standard subscription processing time |
Use for all analysis of Transfer Center Data | Use for historical analysis of all data since impementation | Use for historical analysis of throughput data within last 2 years | Use for current operations status |
How is the performance of our Transfer Center? | How much have we improved since implementation? | How have we been trending over the last 2 years? | What’s happening right now? |
The table below shows each lane of data available for our data sources
When is this happening?
TeleTracking is targeting early 2024 for the deployment of extracts across all customers.