Problems With the ACAOM Second Resolution and Recommendations

Why did ACAOM decide at its August 2009 meeting to revisit a February 9, 2008 resolution on the development and piloting of accreditation standards for accrediting first-professional doctorate programs in acupuncture and in Oriental medicine, as stated in its Second Resolution on First-Professional Doctoral Standards?Why did it adopt a motion that would “continue the comment period for seeking information and consensus regarding the first professional doctorate until January 15th, 2010”?Based on what comments and from what stakeholders?How was this decision reached, or more correctly, why was this resolution passed?

 

The February 2008 Resolution On First-Professional Doctorate Standards by ACAOM succinctly laid down the background of the evolution of the first-professional doctorate.  The net result of this process was the declaration by ACAOM that there was at that time “insufficient evidence of consensus within the acupuncture and Oriental communities of interest…to warrant implementing a first-professional doctorate as entry-level into the profession.”ACAOM resolved that the AOM community should continue to seek consensus “by whatever means deemed appropriate regarding the issue of a first-professional doctorate as entry-level into the profession” and that once such consensus is reached, “the Commission will renew its efforts to develop and subsequently pilot standards, policies and procedures for first-professional doctoral programs in AOM as entry-level into the profession.”So this leads to the important, but as yet, unconsidered questions regarding the FPD posed in the opening paragraph.What comments did ACAOM receive, from whom were they from and what qualified this input as sufficient to reopen the comment period?It is my contention that there was a sufficient lack of evidence from the practitioner, student and educational communities to reopen the comment period that we are in now.I assert that if the evidence I suspect is behind the ACAOM motion to continue comment on the FPD, then ACAOM should repeal its motion to continue the comment period regarding the FPD as this "evidence" is borderline disingenuous in its presentation and has major sampling issues that make the data contained therein unreliable as a gauge of support for the FPD.

 

First we must consider the population that composes the stakeholder community.The U.S. Secretary of Education defines this community in 34 CFR 602.13 as “educators and educational institutions, licensing bodies, practitioners and employers in the professional…fields for which the educational institutions or programs within the agency’s jurisdiction prepare their students.” According to a letter written in early 2009 by Mark Seem, the president and founder of the Tri-State College of Acupuncture and a vocal proponent of the FPD, “…the overall number of practitioner respondents to AAAOM’s December 2008 survey on the FPD is far higher, and represents significantly greater support for the FPD for entrance into the AOM field, than ACAOM’s initial survey in 2003.”Seem argues that the “survey provides sufficient evidence of consensus from the practitioner community for ACAOM to renew its review of the draft FPD standards.”Further, in addition to a Council of Colleges motion in May of 2008 which supported the development of appropriate FPD standards and supported a renewal of ACAOM’s review of those standards, Seem posits that there is sufficient evidence from all stakeholders for ACAOM to do just that.This letter/position paper by Seem appears to address ACAOM directly, likely intended for the ACAOM Commissioners.It exists through the Dragon Rises website, accessible by a Google search.The date is in the title of the document Google created during a web crawl, which dates its composition at 1.11.09.The ACAOM Commissioners met in Columbia, Maryland for their winter Strategic Planning Session over the weekend of February 6-8th.It is possible that the ACAOM considered Seem’s position, but decided to wait until the AAAOM survey was complete before considering reopening the comment period.Indeed, Seem states that the preliminary data he is privy to is "sufficient for ACAOM to renew its review of the draft FPD standards so that appropriate FPD standards might be developed and adopted in a timely fashion" and he asks that "ACAOM consider the adoption of these draft standards now, so that they might enter the trial criteria process and in order for colleges that so choose to begin the final processes necessary for piloting these FPD programs."  The AAAOM survey ended on May 13th.

 

Following the chronology of the FPD timeline, it is reasonable to postulate that the ACAOM Second Resolution on First Professional Doctorate Standards in August of 2009 was adopted as a direct result of the line of reasoning outlined by Seem in his January position paper.The ACAOM resolution states, “Based on comments received from a number of stakeholders in response to the Commission-adopted resolution, ACAOM adopted a motion at its summer 2009 meeting to continue the comment period for seeking information and consensus regarding the first professional doctorate until January 15th, 2010.”This is, of course, where we are at now.And up until this point, no one has questioned exactly what “comments” were received and from which “stakeholders”.And that is what I am asking here.Because if the commentary is essentially the AAAOM survey conducted from November of 2008 until May of 2009, then the ACAOM must consider immediately rescinding the Second Resolution that reopened the comment period.The AAAOM survey is problematic in several critical aspects.It has an inferior sampling methodology and takes liberties in its data presentation.It also does not show that there is “significantly greater support for the FPD as entrance into the AOM field, than ACAOM’s initial survey in 2003” as Seem proclaims.Lacking this, and fundamental flaws in its sampling method, the comment period should have never been reopened.At the very least, the information presented here must be considered by the ACAOM Commissioners at their winter meeting in February and in a fashion similar to its 2008 resolution, they should table the doctorate once again.

 

AAAOM SURVEY

 

First off, the way that the AAAOM presents their survey information up front is borderline disingenuous.Summarizing the survey data, the report states, “For the FPD in acupuncture, of those who expressed an opinion 65% were supportive and 38% were not supportive.”Immediately, we have a major problem with the way the data is being presented.The only way a substantial majority (65%) of respondents “support” the FPD is if you consider a subgroup of the total sample: “those who expressed an opinion”.What does this mean?Not only do the figures presented at the outset leave out those that had “No Opinion”, they also leave out the people who happened to be neutral or undecided- the “Neithers”.If one looks at Appendix A, they will see very quickly how the real numbers add up.55%, as opposed to 65%, actually support the FPD in acupuncture from all survey categories- administrators, students and practitioners.There is a majority, but a slimmer and less impressive one.Now remember, this is for all categories and Dr. Seem was claiming that there is significantly greater support for the FPD from practitioners with the AAAOM survey than with the 2003 ACAOM survey.In that survey, 38.2% of practitioners showed any support for the doctorate, though it appears that the survey is assessing support for both acupuncture and for Oriental medicine, not just acupuncture.And indeed, the AAAOM survey summary reports that 57% of practitioners showed any support for the FPD in acupuncture.That’s a leap of nearly 20%.However, the numbers being displayed up front are only of those who expressed an opinion.When including the “Neither” and “No Opinion” data, the real number is 48% practitioner support, less than an overall majority.This means that the difference is actually only about 10% from the ACAOM 2003 survey.This increase is much less “significant.”In short, a majority of practitioners surveyed by the AAAOM did not support the FPD in acupuncture.All the categories can be assessed this way and all the support numbers come down from what is presented in the page 2 summary for all respondent categories, for both acupuncture as well as for Oriental medicine.

 

The real numbers do not indicate the support that the survey summary cheerily reports for the doctorate in acupuncture or in Oriental medicine.  The finer distinctions of this data were not made very clear by the hard as nails reporting of Acupuncture Today.Yet, even with weaker support than advertised by AT, the AAAOM and presumably reported to ACAOM at their Winter and Summer 2009 meetings, the real problem isn’t with these numbers, it is with the data, or more specifically, how it was gathered.

The AAAOM survey, and likely its ACAOM predecessor, is a non-random sample.It was primarily distributed through e-mails, internet forum posts, and website mentions.What this means, is that there is no way to determine whether those that completed the survey actually represent general opinion in the field.The survey is on the small side, with 662 practitioners responding out of nearly 30,000. But again, the real problem is that the survey is not randomized properly in a probability sampling scheme.With a probability sampling scheme "is one in which every unit in the population has a chance (greater than zero) of being selected in the sample, and this probability can be accurately determined. The combination of these traits makes it possible to produce unbiased estimates of population totals, by weighting sampled units according to their probability of selection."  Examples include Simple Random Sampling, Systematic Sampling, Stratified Sampling, Probability Proportional to Size Sampling, and Cluster or Multistage Sampling; any could have been used to accurately determine the presence or lack thereof of support for the FPD amongst stakeholders. 

A truly indicative sample of support from the AOM practitioner community would have the names of all AOM practitioners and select a random person from that population.These selected individuals would then be interviewed.This kind of sampling can be expensive and time consuming, but gives a more accurate assessment of the population.So, it is impossible to generalize the 662 practitioner responses in the AAAOM survey to the nearly 30,000 licensed practitioners in the US.Hence, the AAAOM survey should be taken with a large grain of salt- treated cautiously and not taken too seriously.But it appears that the ACAOM may have done just the opposite and reopened the comment period on what amounts to an internet survey.And while it is conceded that random sampling is expensive and time consuming, so is the doctorate degree that this survey is opening up the door to.And accordingly, the resources need to be gathered by whatever means are appropriate to determine actual support for the doctorate from a random sample of the entire AOM community before the standards may continue to be drafted and programs may be piloted.But again, this needed to happen before the comment period reopened.In any case, ACAOM must consider the claims of the AAAOM survey as being representative of general practitioner opinion as baseless and cease the drafting of standards and piloting of the doctorate.Of course, the same can be said for all categories of opinion in the survey.Random sampling should be done for students and administrators as well.Until this happens, any claim of opinion of entire populations of stakeholders should be seen for what they are- speculative and potentially subject to bias.

 

As a matter of fact, there is a high likelihood that the both practitioner and student opinion represented in the AAAOM survey is overrepresented by the Tri-State College of Acupuncture student body and alumni.In Seem’s letter to ACAOM, he states that the “AAAOM survey developers were unsuccessful in soliciting cooperation from AOM colleges to help AOM students complete this survey, probably because this request came when most AOM colleges were in exam week or on winter break already.”Yet, Tri-State conducted a survey of its own student and alumni population from November 18th to 28th. The AAAOM survey commenced on November 21st. The Tri-State survey gathered 83 student responses and 148 practitioner responses.The total response count for students on the AAAOM survey was 208.Practitioners were tallied at 662.It is reasonable to theorize that AAAOM survey announcements were able to find their way to the Tri-State student body, being that their own survey started at the same time and likely did not coincide with exams, as well as the obvious strong opinions of the schoolspresident, Dr. Seem.As a result, there is a strong possibility that the opinion of the students and alumni/practitioners of Tri-State are heavily overrepresented in the AAAOM survey.This phenomenon is known as selection bias.Wikipedia defines selection bias as a “statistical bias in which there is an error in choosing the individuals or groups to take part in a scientific study.It is sometimes referred to as the selection effect.The term “selection bias” most often refers to the distortion of a statistical analysis, resulting from the method of collecting samples.If the selection bias is not taken into account then any conclusions drawn may be wrong.”(emphasis mine)If the survey was done in a fashion that is acceptable to statisticians, this information could be either confirmed or refuted.It is assumed that the ACAOM has the names of all respondents to check for this error.But even if it wasn’t the case and Tri-State was not accounting for a disproportionately higher volume of AAAOM survey responses, this survey would still be non-random and likely not representative of the greater AOM population, whether the opinion of students, practitioners or administrators.Even still, the Tri-State FPD survey is also non-random.

 

As a result of the manner in which the survey was distributed, there are other characteristics that may be shared by those that completed the survey that aren’t necessarily representative of the entire field of practitioners or students for that matter.These characteristics may include; recent graduates, people that own a computer, people with a current e-mail address, people that frequent online forums, those that participate in online discussion boards, and AAAOM members.These categories are speculative, though based on the distribution strategies they are reasonable.It is great that the survey designers stratified by AAAOM membership, however random sampling was not done within each strata. Also, it seems that many acupuncturists do not participate in online discussion boards, especially those that do not practice for whatever reason.These groups would be underrepresented in the population sample and may have strong feelings about the issue.

 

These speculations are not meant to be presented as a matter of bad faith on the survey designers, or Dr. Seem for that matter.Nor is the allegation that ACAOM may have reumed the comment period over faulty data meant to give them a black eye.  Those who designed the survey likely did everything in their power to get a higher response rate.The problem isn’t the survey questions or the survey designers.The problem is with the methodology and the ensuing data.

 

The AAAOM survey analysis is therefore not rooted in solid data.The survey may suggest “majority support for FPDs by the AOM profession”, but it does so only if one discounts the most basic elements of sound sampling methodology.Any results and interpretation of the AAAOM survey data and ensuing comparison to the ACAOM 2003 survey data are highly speculative and are should not have been used as evidence for ACAOM to reopen the comment period for the FPD.It is strongly suggested that the ACAOM rescind its Second Resolution if the AAAOM survey data is the primary reason for continuing the comment period of the FPD.As the data is highly questionable due to sampling issues, it cannot serve the function ascribed to it by Mark Seem; it is not indicative of a significant show of approval from students or practitioners of AOM.If this is not the data that lead to a continuance of the commentary period concerning the FPD, then it is the responsibility of the ACAOM to reveal exactly what stakeholder information reopened the comment period.It is also strongly suggested that the AOM profession go about conducting a census of the profession through the appropriate professional organization(s).A proper random sample of the licensed acupuncturist population is necessary to make inferences about the opinions of the larger group.Without this step, any further action taken by the ACAOM allowing for the development of doctoral draft standards and subsequent pilot programs will never be able to demonstrate that these procedures are ‘widely accepted” by the AOM stakeholder community, and therefore short of the requirements in USDE 34 CFR 602.13.

*much appreciation to both Dr. Kim Tippens of the Helfgott Research Institute and an anonymous disenchanted PhD student on the verge of leaving academia for a life of travel writing using my questions to procrastinate on her never-going-to-happen dissertation with guidance and clarity on these issues.

This story was posted on January 4 2010 by LarryG.
Tags: acaom survey

Comments

  • January 4 2010 at 2:12 PM
    Guest writes:

    AOM Community Forum postings: “We are owed an explanation”

    Hmmm…I’m so kung fu tze (confucious, confused).

    The FPD powers that be seem to not fully grasp what it is they’re trying to accomplish either.

    It would seem (ode to you know who…haha) that the discussion on the Professional Doctorate (not a “first” professional doctorate to ms hobbs) is being muddled with semantics.  Thanks to Matt Bauer for demanding clarification.

    to sum it up:
    “Entry into the profession could happen at either the Masters or Doctorate
    level. ” (jabbour).

    Mr. Jabbour even sought clarification for ACAOM with regards to semantics surrounding the Professional Doctorate as its intention was not to “replace the entry level masters” and is not, in fact, a “first” professional doctorate.

    I see the veils are coming down, and it really is about ego stroking.  So what is it really?  And why have to different, yet “seemingly equal” entry points to unlimited licensure (or is there something dubious in future state legislation that may come of it?

    Respect from other health professions?  Not so much if the “powers that be” cain’t (intentional misspelling) even figgerit out.

    Going back to Jabbours initial interpretation that entry to the trade (cuz that’s what it is—regional vs national accreditation speak) could happen at the masters or the doctorate.

    To put it in inner city hospital unit vernacular: “Honey, that don’t make no kinda sense atall!”

      0 likes
  • January 4 2010 at 2:26 PM
    Lisafer writes:

    this post is a thing of beauty

    not least because in the midst of this incredibly muddled discussion, I also managed to lose track of the extremely important question you just put back into the center, where it belongs:

    Wait, why are we talking about this again? No, really—WHY ARE WE TALKING ABOUT THIS AGAIN? 

    Thank you Larry, thank you Kim, thank you anonymous Ph.D student.

    “You know how people always say there’s a reasonable explanation for things like this? Well, there isn’t.” Daniel Pinkwater, The Neddiad

      0 likes
  • January 4 2010 at 4:03 PM
    Nora writes:

    shut em down, LG!

    I just love it when someone from this anti-intellectual group of scallawags uses statistics to counter the FPD.  Here’s hoping it works!

      0 likes

Add your 2 cents!

You must be logged in to comment on the blog.


Who Likes This:

Featured Events

-- placeholder --

Who's been POCAing lately.

  • OpenSourceEverything commented on the blog post 'New POCA Tech FAQs' in the blog 'Prick Prod and Provoke Blog' - Wednesday, March 13, 2019
  • urbanapuncture commented on the blog post 'Fatphobia Part 3: Some Thoughts about Diet Culture and Talking to Patients' in the blog 'Prick Prod and Provoke Blog' - Wednesday, March 13, 2019
  • Roppy commented on the blog post 'The Acupuncture Profession has 99 problems, but NADA ain’t one of them.' in the blog 'Prick Prod and Provoke Blog' - Tuesday, March 12, 2019
  • OpenSourceEverything commented on the blog post 'Open Sourcing Modern Acupuncture, Part Two: Trouble' in the blog 'Prick Prod and Provoke Blog' - Monday, March 11, 2019
  • OpenSourceEverything commented on the blog post 'Hierarchy of Needs: Too Sexy for This Work?' in the blog 'Prick Prod and Provoke Blog' - Monday, March 11, 2019
  • urbanapuncture commented on the blog post 'Hierarchy of Needs: Poor People's Priorities' in the blog 'Prick Prod and Provoke Blog' - Wednesday, February 20, 2019
  • Lisafer commented on the blog post 'Stone Soup and Social Containers' in the blog 'Prick Prod and Provoke Blog' - Wednesday, February 13, 2019
  • Nityamo commented on the blog post 'POCA Clinic Member Changes' in the blog 'Prick Prod and Provoke Blog' - Wednesday, February 13, 2019
  • weedyknees commented on the blog post 'Stone Soup and Social Containers' in the blog 'Prick Prod and Provoke Blog' - Wednesday, February 13, 2019
  • urbanapuncture commented on the blog post 'Stone Soup and Social Containers' in the blog 'Prick Prod and Provoke Blog' - Wednesday, February 06, 2019
  • swiggs commented on the blog post 'Hierarchy of Needs: Too Sexy for This Work?' in the blog 'Prick Prod and Provoke Blog' - Tuesday, February 05, 2019
  • urbanapuncture commented on the blog post 'Hierarchy of Needs: Too Sexy for This Work?' in the blog 'Prick Prod and Provoke Blog' - Monday, February 04, 2019
  • monkeyacupunk commented on the blog post 'Hierarchy of Needs: Too Sexy for This Work?' in the blog 'Prick Prod and Provoke Blog' - Thursday, January 31, 2019
  • Roppy commented on the blog post 'POCA Clinic Member Changes' in the blog 'Prick Prod and Provoke Blog' - Tuesday, January 29, 2019
  • Bob W commented on the blog post 'POCA Clinic Member Changes' in the blog 'Prick Prod and Provoke Blog' - Thursday, January 24, 2019
  • Spartacus commented on the blog post 'POCA Clinic Member Changes' in the blog 'Prick Prod and Provoke Blog' - Thursday, January 24, 2019
  • mollyfread commented on the blog post 'POCA Clinic Member Changes' in the blog 'Prick Prod and Provoke Blog' - Thursday, January 24, 2019
  • Bob W commented on the blog post 'POCA Clinic Member Changes' in the blog 'Prick Prod and Provoke Blog' - Wednesday, January 23, 2019
  • mcandela55 commented on the blog post 'Hierarchy of Needs: Poor People's Priorities' in the blog 'Prick Prod and Provoke Blog' - Wednesday, January 23, 2019
  • mcandela55 commented on the blog post 'Hierarchy of Needs (for Acupuncture)' in the blog 'Prick Prod and Provoke Blog' - Wednesday, January 23, 2019
  • mcandela55 commented on the blog post 'Open Sourcing Modern Acupuncture: Part Four, Stamina' in the blog 'Prick Prod and Provoke Blog' - Wednesday, January 23, 2019
  • ewolfk commented on the blog post 'POCA Clinic Member Changes' in the blog 'Prick Prod and Provoke Blog' - Wednesday, January 23, 2019
  • NicoleManiez commented on the blog post 'Hierarchy of Needs: Poor People's Priorities' in the blog 'Prick Prod and Provoke Blog' - Tuesday, January 22, 2019
  • Roppy commented on the blog post 'Hierarchy of Needs: Poor People's Priorities' in the blog 'Prick Prod and Provoke Blog' - Thursday, January 17, 2019
  • swiggs commented on the blog post 'Hierarchy of Needs: Poor People's Priorities' in the blog 'Prick Prod and Provoke Blog' - Thursday, January 17, 2019
  • Roppy commented on the blog post 'The Acupuncture Profession has 99 problems, but NADA ain’t one of them.' in the blog 'Prick Prod and Provoke Blog' - Thursday, January 17, 2019
  • Moses.C commented on the blog post 'Hierarchy of Needs: Poor People's Priorities' in the blog 'Prick Prod and Provoke Blog' - Monday, January 14, 2019
  • OpenSourceEverything commented on the blog post 'Open Sourcing Modern Acupuncture: Part Four, Stamina' in the blog 'Prick Prod and Provoke Blog' - Monday, January 14, 2019
  • teatree commented on the blog post 'Open Sourcing Modern Acupuncture, Part Three: Hope' in the blog 'Prick Prod and Provoke Blog' - Sunday, January 13, 2019
  • Presence_CA commented on the blog post 'Hierarchy of Needs: Poor People's Priorities' in the blog 'Prick Prod and Provoke Blog' - Saturday, January 12, 2019
  • CPMike commented on the blog post 'Open Sourcing Modern Acupuncture: Part Four, Stamina' in the blog 'Prick Prod and Provoke Blog' - Thursday, January 10, 2019
  • ChrisRogers commented on the blog post 'Hierarchy of Needs (for Acupuncture)' in the blog 'Prick Prod and Provoke Blog' - Thursday, January 10, 2019
  • whitney commented on the blog post 'Hierarchy of Needs (for Acupuncture)' in the blog 'Prick Prod and Provoke Blog' - Wednesday, January 09, 2019
  • keithananda commented on the blog post 'Open Sourcing Modern Acupuncture: Part Four, Stamina' in the blog 'Prick Prod and Provoke Blog' - Wednesday, January 09, 2019
  • ellengrover commented on the blog post 'Hierarchy of Needs (for Acupuncture)' in the blog 'Prick Prod and Provoke Blog' - Tuesday, January 08, 2019
  • tatyana commented on the blog post 'Hierarchy of Needs (for Acupuncture)' in the blog 'Prick Prod and Provoke Blog' - Tuesday, January 08, 2019
  • Moses.C commented on the blog post 'Hierarchy of Needs (for Acupuncture)' in the blog 'Prick Prod and Provoke Blog' - Monday, January 07, 2019
  • Jess B commented on the blog post 'Hierarchy of Needs (for Acupuncture)' in the blog 'Prick Prod and Provoke Blog' - Monday, January 07, 2019
  • ewolfk commented on the blog post 'Hierarchy of Needs (for Acupuncture)' in the blog 'Prick Prod and Provoke Blog' - Monday, January 07, 2019
  • Justine commented on the blog post 'Public Service Announcement: POCA and Ear Acupuncture!' in the blog 'Prick Prod and Provoke Blog' - Thursday, January 03, 2019
  • sivesind commented on the blog post 'Public Service Announcement: POCA and Ear Acupuncture!' in the blog 'Prick Prod and Provoke Blog' - Sunday, December 30, 2018
  • Lisafer commented on the blog post 'Open Sourcing Modern Acupuncture, Part Three: Hope' in the blog 'Prick Prod and Provoke Blog' - Friday, December 28, 2018
  • Lisafer commented on the blog post 'Here's to Open Sourcing: Modern Acupuncture, Part One' in the blog 'Prick Prod and Provoke Blog' - Friday, December 28, 2018
  • keithananda commented on the blog post 'Open Sourcing Modern Acupuncture, Part Three: Hope' in the blog 'Prick Prod and Provoke Blog' - Thursday, December 27, 2018
  • keithananda commented on the blog post 'Open Sourcing Modern Acupuncture, Part Three: Hope' in the blog 'Prick Prod and Provoke Blog' - Thursday, December 27, 2018
  • Roppy commented on the blog post 'Open Sourcing Modern Acupuncture, Part Three: Hope' in the blog 'Prick Prod and Provoke Blog' - Wednesday, December 26, 2018
  • Roppy commented on the blog post 'The Acupuncture Profession has 99 problems, but NADA ain’t one of them.' in the blog 'Prick Prod and Provoke Blog' - Tuesday, December 25, 2018
  • crismonteiro commented on the blog post 'Open Sourcing Modern Acupuncture, Part Three: Hope' in the blog 'Prick Prod and Provoke Blog' - Tuesday, December 25, 2018
  • ewolfk commented on the blog post 'Open Sourcing Modern Acupuncture, Part Two: Trouble' in the blog 'Prick Prod and Provoke Blog' - Monday, December 24, 2018
  • Spartacus commented on the blog post 'Open Sourcing Modern Acupuncture, Part Two: Trouble' in the blog 'Prick Prod and Provoke Blog' - Sunday, December 23, 2018