Hands-on RAG for Education:
From a Generic Chatbot to
a Document-Grounded System

Lab Test Questions and Course Advising Examples

Tutorial at the 27th ACM Annual Conference on Cybersecurity and Information Technology Education
(ACM SIGCITE 2026), Wilmington, North Carolina, November 12–14, 2026
Tutorial date and time: to be announced

ACM SIGCITE logo ACM SIGCITE 2026 conference logo

Test Questions

Back to tutorial materials

Part 1. Testing Set for Lab Practice

The lab corpus has two documents, the ITM Graduate Student Handbook (ITM_Student_Handbook.pdf, Fall 2025) and the Tuition, Fees, and Costs page (Tuition__Fees__and_Costs.html, 2026–27 estimates). Run each question through all four approaches and score every answer from 0 (fails) to 2 (meets) on retrieval relevance, grounding, citation accuracy, absence of unsupported claims, and correct refusal when the documents do not hold the answer.

Q1Graduate tuition

What is the tuition rate for a graduate student?

Expected answer

For a new graduate student at the Mies Campus, estimated tuition for the 2026–27 academic year is $34,650, based on 18 credit hours per year. Fees are $1,850, and the total estimated cost of attendance is $59,972.

Source

Tuition__Fees__and_Costs.html, “New Graduate Student (Mies Campus)” table.

Challenges to answer this question
  • This is the baseline question, and every approach should answer it correctly.
  • The answer sits in a table, so document conversion must keep each number attached to its row label.
Common mistakes in answers
  • Reporting the undergraduate tuition of $53,198 as graduate tuition.
  • Reporting the total cost of attendance, $59,972, as tuition.
  • Leaving out that the estimate assumes 18 credit hours per year.

Q2Tuition for an international student

What is the tuition rate for an international student?

Expected answer

The tuition page does not list separate rates for international and domestic students. Its estimates depend on the program level instead. For 2026–27, estimated tuition is $53,198 for a new undergraduate, $34,650 for a new graduate student at the Mies Campus (based on 18 credit hours per year), and $58,435 for a new JD student. The assistant should ask which program the student is in and point to the Student Accounting Office website for detailed rates.

Source

Tuition__Fees__and_Costs.html, the three cost-of-attendance tables and the “Components” section.

Challenges to answer this question
  • The page separates tuition by program level (undergraduate, graduate, JD), not by international or domestic status.
  • The question does not say which program level the student is in.
  • A good answer states that the page has no international rate, then lists the rates by level or asks which program applies, instead of giving a single number.
Common mistakes in answers
  • Inventing an “international student rate.”
  • Giving the graduate or undergraduate figure alone as the international rate.
  • Presenting general costs, such as health insurance, as charges specific to international students.

Q3Problems with Canvas

I am having problems with Canvas. Who should I contact, and how?

Expected answer

Contact the OTS Support Desk at supportdesk@illinoistech.edu or 312.567.DESK (3375). ITM faculty, teaching assistants, and staff cannot help with Canvas issues.

Source

ITM_Student_Handbook.pdf, page 27, “Canvas and Online Courses.”

Challenges to answer this question
  • The email address and phone number must be reproduced exactly as written.
  • The sentence that follows the contact details limits who can help, and a complete answer includes it.
  • The handbook names many contacts for other situations, so passages retrieved from nearby sections can point to the wrong person.
Common mistakes in answers
  • Telling the student to contact the course instructor or the ITM Program Manager.
  • Altering the email address or phone number. The source gives the number both as 312.567.DESK and as 3375.
  • Leaving out that ITM faculty and staff cannot help with Canvas.

Q4Taking two online courses

Can I take two online courses this semester?

Expected answer

It depends on visa status. Students on F-1 visas may take only one online course per semester, a U.S. government requirement that cannot be waived, and F-1 students cannot enroll in online sections of any course in their first semester. The handbook states no comparable limit for students who are not on F-1 visas, although students in live sections are still expected to attend class in person.

Source

ITM_Student_Handbook.pdf, page 28, “ITM Online Course Policies.”

Challenges to answer this question
  • The correct answer depends on the student's visa status, which the question does not give.
  • Without a student profile, a good answer separates the cases or asks about visa status. With the profile “I am an F-1 international student in my first semester,” the answer becomes that the student cannot take any online course this semester.
  • The source spells F-1 as “Fl Visa,” with a lowercase letter l. A keyword search for “F-1” may miss the passage, while dense retrieval can still find it.
Common mistakes in answers
  • Answering yes or no without stating the condition on visa status.
  • Missing the rule that first-semester F-1 students cannot take online courses at all.

Part 2. Example Questions in a More Complicated Task

Course recommendation goes beyond document question answering. The two questions below draw on the PhD in Information Technology catalog page (PhD_in_IT.html), the ITM course descriptions (ITM_Course_List.pdf), and the handbook's term planning list. Retrieval supplies the rules, but an answer also needs filtering, reasoning, and the student's own record.

C1Finishing the PhD core requirement

I am starting the ITM PhD with a Master of Information Technology and Management from Illinois Tech. In my master's I completed ITMD 514 (A), ITMD 522 (A), ITMO 540 (B), ITMO 556 (B), and ITMS 548 (C). Which courses could I take this spring to finish my core requirement?

Challenges to answer this question
Rules to apply
  • Students entering with a master's in IT must take at least three courses from three different core groups.
  • A course meets a core requirement only if it was taken at Illinois Tech and passed with B or better.
  • Each course has its own prerequisites, some with a minimum grade.
  • Some courses are normally offered only in the fall.
Filtering
  • Remove core groups already met. ITMD 522 meets Group 5. ITMS 548 is in Group 4 but has a C, so Group 4 is still open.
  • Remove courses whose prerequisites the student has not completed, such as the Group 1 courses that require ITMD 510.
  • Remove courses that are normally fall-only, such as ITMO 544 and ITMS 528.
Reasoning
  • The student still needs two more courses, and they must come from two different groups among Groups 1, 2, 3, 4, and 6.
  • The eligible set is a set operation over courses, prerequisites, terms, and the student's record.
Eligible = ( ∪ Courses(g) for g not in {5} ) ∩ { c : prerequisites of c are met } ∩ Offered(spring) − Completed with B or better
  • The final step picks two eligible courses whose groups differ.
Why retrieval alone falls short
  • The group membership of 36 courses comes from one table, the prerequisites are spread across dozens of chunks in another document, and the term information sits in a third.
  • Two different grade thresholds apply, B or better for core credit and minimum grades for some prerequisites.
  • The source data contains errors. ITMO 541 lists “ITMD 556” as a prerequisite, which is most likely a typo for ITMO 556, and a rule engine that copies it would wrongly exclude the course.

C2Transfer credit and remaining requirements

I have an MBA from another university (36 graduate credits, all with grades of B or better). As a non-degree student at Illinois Tech, I also completed 9 credits of 400-level ITM courses. If I join the ITM PhD, how many credits can I transfer, how many more must I complete, and can any of these courses satisfy core requirements?

Challenges to answer this question
Rules to apply
  • The catalog defines three entry tracks with different numbers. For a master's degree outside IT or computing, the degree needs at least 72 credits, allows at most 23 transfer credits, and requires 49 credits at the 500, 600, or 700 level.
  • 400-level ITM courses taken at Illinois Tech cannot be applied to the degree.
  • Core courses must be taken at Illinois Tech, so transfer courses cannot meet core requirements.
  • This track requires one course from each of Groups 1, 2, and 3, two courses from two of Groups 4, 5, and 6, and three sequential semesters of ITM 695.
Filtering
  • Identify the correct track among the three, since an MBA is a master's outside IT.
  • Exclude the 9 credits of 400-level Illinois Tech ITM courses from transfer.
  • Exclude all MBA courses from core credit.
Reasoning
Transfer = min(23, 36) = 23 credits Remaining ≥ 72 − 23 = 49 credits, all at the 500 level or above Core = choose courses covering Groups {1, 2, 3} and at least two of Groups {4, 5, 6}
  • The total may exceed 72 credits, because the adviser can require prerequisite courses and undergraduate-level prerequisites do not count toward the degree.
Why retrieval alone falls short
  • The rules are spread across three requirement tables, numbered footnotes, and a separate notes section, and missing any one of them produces a wrong number.
  • The answer needs arithmetic and a set-coverage condition, not a passage to quote.
  • The source is inconsistent with itself. The table lists the research course as ITM 691, while a footnote calls it ITMT 691.