This course is meant to provide the basics of what Analytics Engineers need to know about Compass to have it help them in their work. It will cover Explorer and Diagnostics, but not Opportunity Scan or Presentations.
This page is meant to be used while the reader is actively working inside Compass, so make sure that you have access to a Compass environment before continuing.
The Compass Product Manual is a comprehensive reference to the tool, which you should use as a complement to this training course.
1. Explorer
The landing page of Compass is the Explorer "New Session" tab, which has an interface that will be familiar to anyone who has used ChatGPT or similar tools.
Enter
What is the row count of ursa.mi_aa_004?
into the textarea and press the Start button.
You will be immediately be brought into the Session screen.
2. Sessions
Sessions are the workhorse that power much of the functionality in Compass. Sessions represent a single context window for an interaction with an agent. That is to say, the model has no other instructions than what you can see in this session screen.
The main panel of the session screen is rendered as a three-way chat between the agent, the Compass software, and the user. A key part of learning how to use Compass responsibly is to be able to read this chat. Happily, it's typically fairly easy to follow.
By this point, Compass has probably already answered your question (it's 100,000), but scroll to the top of the chat window to see how it got there.
At the beginning of the chat is an enormous block of text that starts with several screenfuls of best practices. This is coming from the Diagnostic Prompt of the default Explorer Persona. We'll get to that shortly. Next is a description of all the tools that Compass has made available to the agent in this session, such as QUERY_DATABASE. These are also defined in the Explorer Persona, as we will see.
At the bottom of the large initial chat bubble is the specific question that you asked.
Now the ball is in the court of the agent, and the agent acts by using one (or more) of the available tools. The agent uses its best judgment about which tools to use; this is a hallmark of agentic AI.
The agent will probably either call GET_OBJECT_METADATA on the table in question, or it might jump straight to QUERY_DATABASE.
The next chat bubble will be the response of the tool call, as generated by procedural Compass code. You can expand the bubble to see the full response. Note that the table response from the QUERY_DATABASE call represents the actual response from the database, uninterpreted by the agent.
Eventually, the agent uses the PROMPT_USER tool call to return control to the user. If the agent had skipped over GET_OBJECT_METADATA and you're interested to see what that tool response looks like, you can go ahead and ask
Can you run GET_OBJECT_METADATA on that table for me?
as a follow-up. You should see the metadata for this object, along with the SQL that built the table. Indeed, if you don't see the SQL, talk to your Ursa Health CSM, as this important functionality may need to be enabled in your environment.
3. Findings
When control is returned to the user, ask this follow-up question:
Can you promote the row count as a finding?
The PROMOTE_FINDING tool allows the agent to lock in any QUERY_DATABASE response from the session as a worthwhile artifact.
The finding should appear on the rightbar, and you can expand it by clicking on it. The table of data (this one only has one cell) and the SQL (which you can see by clicking the SQL button) are historically accurate and hallucination-free. The title and the annotation are created by the agent, but can be edited via the edit button.
This finding is a quite boring one, which is why the agent chose not to put it in a chart. Press the "Collapse Findings" button to return to the session screen, and ask a slightly more interesting question.
Can you promote me a bar chart of count of records by month, ordered by count desc?
The agent will probably query the database to get the result, and go ahead and promote the result, before returning control to the user. When returning control to the user, it will probably summarize some of the numbers. The agent is usually does this correctly, but the a general rule of thumb is, the left-hand side of the session page is prone to hallucination, where as the right-hand (findings) side is cold hard fact. In this case, both the single-value row count and the bar chart are the two findings on the right-hand side.
4. The FULL_SEARCH_DATA_MODEL tool
Click back on the Explorer link on the leftbar to open a new session. This time, enter in a slightly less precise question, as follows:
What is the row count of the calendar dates table?
Keep in mind that each session is a separate context window, so it will have retained nothing from the previous session about this table.
You will likely see the agent thrash around, trying to find this table, because this object is poorly documented in the data model. Indeed, you might have noticed in step 2 that the object's description isn't set.
The FULL_SEARCH_DATA_MODEL tool will probably return results for some of the search terms during this thrash, but the agent will recognize those objects as not being relevant. The agent might even try to guess at the table name with some QUERY_DATABASE tool calls.
It's important to be able to recognize agent thrash, because it's an indicator that the agent is suffering and will likely perform sub-optimally. In this case, hopefully the agent will report back its failure to the user, but with a less verbose Diagnostic Prompt the agent might start confidently return nonsense results.
5. Explorer Previous Sessions tab
Click back on Explorer in the leftbar, and click on the "Previous Sessions" tab. This where you can see a list of your previous sessions for Explorer specifically. As we'll see, sessions are a building block not just for Explorer but for other parts of Compass as well.
6. Explorer Personas
If you read the large Diagnostic Prompt at the start of the chat session you might have noticed that it was very opinionated, and that its opinions were a better fit for an analyst than an analytics engineer. Indeed, the default Explorer Persona is the Analyst Assistant, but this is configurable for each user.
Navigate to Diagnostics via the leftbar. Depending on the maturity of the environment there might be many or few diagnostic definitions available. The diagnostic definitions are organized by domain. Scroll down to the EXPLORER domain and click on the Analyst Assistant diagnostic.
You'll see here the large system prompt that powered the first chat bubble of your first Explorer sessions. This is where that came from. The available tool calls are enumerated below, and beneath that you'll see that there are no pre-defined sessions.
Having no pre-defined sessions is a hallmark of an Explorer diagnostic, because these sessions are presumed to come ad-hoc from the user each time. In this way we can see that Explorer is built on diagnostics, but does not use the full breath of pre-definition that diagnostics provide.
Back out to the diagnostic screens, and click the star button next to the "AE Assistant" diagnostic. This will add the "AE Assistant" diagnostic as a second available Explorer persona. Indeed, click back to Explorer and you'll see a new "Persona" dropdown that will let you choose the "Persona". All this does is start the session out with the initial prompt and available tools from the diagnostic you select.
7. Creating Diagnostics
Explorer is not the only way to run sessions. As we saw above, Explorer is largely powered under the hood by diagnostics, but it's possible to run these diagnostics directly. Because they're both built of the same session mechanics, the experience is quite similar.
Click on the diagnostics icon in the leftbar and note that the three tabs are called "Definitions", "Instances", and "Suites". Our use of the term diagnostic is ambiguous; it could be referring to the "definition" of the diagnostic, or an instance that's been created from that definition. When mentioning diagnostics for Explorer so far, I've been referring to the diagnostic definition. Under the hood, exactly one diagnostic instance gets created the first time Explorer is used (for each persona). But there are use cases where a user will want to instantiate a single diagnostic definition repeatedly into different instances over time.
Diagnostic Suites are beyond the scope of this tutorial.
Find and click on the "AE Training" diagnostic. Press the "Create Diagnostic" button, which creates an instance. Navigating back to the "Instances" panel, you should see this training instance. If you click the "Include Explorer Diagnostics" checkbox you'll also be able to see the diagnostic instances for your Explorer work. This should underline the fact that Explorer is just diagnostics under the hood. Typically you're going to just access your previous Explorer work through the Explorer tab however.
Unlike the diagnostics powering Explorer personas, the AE Training diagnostic has several session prompts pre-baked. Let's run through them.
Click on the first session, "Agent as AE cowboy", and you'll see that the session prompt is already filled. Press the "Start" button to get the agent going.
If the agent is feeling good, it will notice that this concept has been approached in various Ursa measures for different windows but not for 140 days. It will have already seen the measure SQL in its GET_MEASURE_METADATA tool call, so, it will probably go ahead and do the analytics engineering to adapt that SQL to 140 days.
If you see Compass start in with Analytics Engineering, you should be on high alert. It's often eager to do so, and it's typically the responsibility of the diagnostic-scope prompt to give it the appropriate guardrails. (This training diagnostic does not have this responsible set of guardrails defined.) For this simple tweak, the agent might be able to get it right, but our house rule of thumb is that the agent is not capable of reliably doing analytics engineering work.
8. Long-term memory vs short-term lookups
Click on the diagnostics leftbar icon to return to the home page for diagnostics. Go to the Instances panel, click on the "AE Training" diagnostic and press the "Edit Diagnostic" button. Do not do this to the "AE Training" diagnostic definition! You should be editing the instance. Unclick the "Search data model" tool and press the "Save Diagnostic" button. This will remove the access to SEARCH_DATA_MODEL from the agent within the continued scope of the sessions of this diagnostic instance. Do not click the "Save as Definition" button, this will create a new reusable definition based on the current state of this diagnostic instance.
Select and run the second session, the one entitled "ICD10 enumeration from long-term memory". It might take some cajoling, but the agent will probably respond with a list of ICD-10 codes. What's going on here is impressive and a little terrifying: Opus basically memorized the internet during its training phase, and has a pretty good recall of all the ICD-10 codes in the same way that it has basically memorized all the sonnets of Shakespeare.
We're going to push the limits of good taste and ask the follow-up instruction:
promote these as a finding
At this point, the agent may create a SQL query with its text hardcoded into SQL as string literals. This breaks the entire premise that findings are supposed to represent the truth straight from the database.
SELECT 'J09' AS icd10_code, 'Influenza due to certain identified influenza viruses' AS description, 'Novel influenza A (avian, swine flu)' AS category
UNION ALL
...
Hopefully, the agent will have some misgivings about promoting this query as a finding, because up in the PROMOTE_FINDING tool description we've explicitly told it not to do that sort of thing.
Click back to the diagnostic and run the "ICD10 enumeration from value sets table" session. In this session, the agent should act the way we'd want it to act; looking up metadata for our value sets table, finding the names of the value sets, and querying for the values based on the appropriate names.
Now when you
promote these as a finding
it should have no qualms in doing so.
Be sure to edit the diagnostic to re-enable "Search Data Model" before moving on.
9. Practical use-case: Logic Tracing
One of the applications that we think will be useful for analytics engineers is Compass's ability to investigate surprising results back into the data model.
The next session replicates the workflow described in https://docs.ursahealth.com/docs/how-to-understand-results-in-analytics-portal.
Follow the thought process of the agent as it calls GET_MEASURE_METADATA on URSA-PHF-R12, then starts calling GET_OBJECT_METADATA on the tables that it saw in the SQL in the GET_MEASURE_METADATA tool response. When it succeeds, it follows much the same logic as documented in our Knowledge Base how-to guide. It eventually tracks back to [SO/URSA-CORE] Primary Care Providers where it determines that if a physician has a history of providing the sorts of service associated with primary care – particularly: preventative care – then they will make our list of primary care providers, no matter what their NUCC code says.