The American Board of Psychiatry and Neurology developed an article-based format for maintenance of certification in three primary specialties and one subspecialty. A 2019 pilot program was assessed with a widely employed validity framework, and the collected data spanned content, response process, internal structure, relation to other variables and consequences. Program enrollment ranged from 66.7% for psychiatrists to 75.3% for child neurologists. Pass rates for the 2019 cohort ranged from 92.6% for child and adolescent psychiatry to 98.7% for neurology; and few diplomates failed or did not complete the process. Psychiatrists had a modest but significant relationship between performance on previous and subsequent maintenance of certification examinations. Ninety percent or more found articles easy to access and helpful to their practices. They also deemed mini-tests a fair evaluation of article knowledge and felt satisfied with the test-taking experience. The diplomates strongly favored the article-based format over traditional multiple-choice examinations. The authors observed the pilot as "a meaningful and relevant learning activity" that benefited patient care.