Getting startedChecklists are always helpful in getting started with experiments, and here's mine, based in large part on Cowart (1997) and discussions with people here at Stanford. I also took notes from Chuck Clifton's intro to experimental design at the 2007 LSA Summer Institute.
Florian Jaeger's notes are very helpful and you should go read those. If, for some reason, you decide that his notes are too long (maybe filler-gap dependencies have shortened your attention span), you can read my summary of his notes here.
You may also want to check out my class notes from Chris Manning's course on "Quantitative, probabilistic, and optimization-based explanation in linguistics" (Autumn 2007).
For introductory stats, I recommend pairing the Baayen and Dalgaard below. (Here are some quick hints of things to keep in mind when you're doing your own stats.)
Finally, here are the notes from Victor Kuperman's more advanced class on building models with R.
My own workAt the end of 2010, I published a paper on using crowdsourcing (Amazon's Mechanical Turk, in particular) for linguistic research--this is joint work with Victor Kuperman.
Learning from othersI've found our lab syntax group to be very helpful in adjusting experimental plans and figuring out how to communicate findings. Browse through these notes from our empirical research seminar.
If you're curious about the rationale for experimentation, you can see this run-down.
grammaticality/acceptability of many different types of sentence seem
I’m interested in looking at whether non-linguists would actually
reject the same sentences linguists have and how this may vary as we
shift factors known to affect processing difficulty. At the core of
this investigation is the sense that people build their understanding
of sentences incrementally and that constraints theorists try to make
the grammar account for may be unnecessary.
Of course, it is only fun to prove that linguists' intuitions are wrong for so long. The important next step is to figure out theories and frameworks that account for the empirical data better.
These are given in "order of attack", rather than alphabetically. There's a heavy amount of sentence processing literature here. For a little less than that, see this list of references (from Cowart (1997)). The citation forms have everything you should need to find the resource, but I haven't been perfectly consistent in my formatting. I hope that doesn't bug you.
Baayen, R. H. Forthcoming. Analyzing Linguistic Data. A Practical Introduction to Statistics. Cambridge: Cambridge University Press. (Still typing up notes on this big, useful introduction to stats for linguistics.)
Bard, E. G., D. Robertson, and A. Sorace: 1996, `Magnitude Estimation of Linguistic Acceptability'. Language 72(1), 32-68.
Cowart, W. (1997) Experimental Syntax: Applying objective methods to sentence judgments. Thousand Oaks, CA: Sage Publications. (A few other notes here and here.)
Dalgaard, P. 2002. Introductory Statistics with R. New York: Springer.
Keller, F. (1999) Review of Carson T. Schuetze's The Empirical Base of Linguistics: Grammaticality Judgments and Linguistic Methodology, Journal of Logic, Language and Information 8: 1, 114--121, 1999.
Reips, U.-D. (2002) Standards for Internet-based experimenting. Experimental Psychology, 49(4), 243-256.
Trueswell, J. C., Tanenhaus, M. K., & Garnsey, S. M. (1994) Semantic Influences on Parsing: The use of thematic role information in syntactic ambiguity resolution. Journal of Memory and Language, 33, 285-318.
Gibson, E., & Pearlmutter, N. (1998) Constraints in sentence comprehension. Trends in Cognitive Science, 2(7), 262-268.
Kay, P. (2005) “Argument Structure Constructions and the Argument-Adjunct Distinction.” In Grammatical Constructions: Back to the Roots M. Fried and H. Boas (eds.) Amsterdam: Benjamins. pp. 71-98.
Sag, Ivan A. and Thomas Wasow "Performance-Compatible Competence Grammar". To appear in Non-Transformational Syntax, edited by Robert D. Borsley and Kersti Börjars.
Wasow, Thomas and Jennifer Arnold "Intuitions in Linguistic Argumentation". Lingua 115.11: 1481-1496. 2005.
Gibson, E. (2003) Sentence Comprehension, Linguistic Complexity in.
Fanselow, G., & Frisch, S. (1999?) Effects of Processing Difficulty on Judgments of Acceptability.
Featherston, S. (2005) Magnitude estimation and what it can do for your syntax: some wh-constraints in German. Lingua, 115, 1525--1550.
Keller, F. (2000) Gradience in Grammar: Experimental and Computational Aspects of Degrees of Grammaticality. University of Edinburgh.
Rohdenburg, G. (2002) Processing complexity and the variable use of prepositions in English. In C. H. & G. Radden (Eds.), Perspectives on Prepositions (pp. 79-100). Tübingen.
Labov, William (1996) When Intuitions Fail. In L. McNair, K. Singer, L. Dolbrin and M. Aucon (eds.), Papers from the Parasession on Theory and Data in Linguistics Chicago Linguistic Society 32: 77--106.
Gordon, P. C., Randall, H., & Johnson, M. (2001) Memory Inference During Language Processing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 27(6), 1411-1423.
Trueswell, J. C. (1996) The role of lexical frequency in syntactic ambiguity resolution. Journal of Memory and Language, 35, 566-585.
Phillips, Colin. 1998 Linguistics and Psycholinguistics: Competence and Performance Systems. Class Notes from The Organization of Language (U. of Delaware).
Fedorenko, E., Gibson, E. & Rohde, D. (in press) The Nature of Working Memory Capacity in Sentence Comprehension: Evidence Against Domain-Specific Working Memory Resources. Journal of Memory and Language.
Gibson, E. (1998) Linguistic complexity: Locality of syntactic dependencies. Cognition, 68, 1-76.
Hawkins, J. A. (2004) Efficiency and Complexity in Grammars. Oxford: Oxford University Press.
Keller, F. (1996) How Do Humans Deal with Ungrammatical Input? Experimental Evidence and Computational Modelling. In D. Gibbon (Ed.), Natural Language Processing and Speech Technology: Results of the 3rd KONVENS Conference, Bielefeld, October 1996. Berlin: Mouton de Gruyter.
Keller, F., & Sorace, A. (2005) Gradience in Linguistic Data. Lingua, 115(11), 1497-1524.
Rohdenburg, G. (1996) Cognitive complexity and increased grammatical explicitness in English. Cognitive Linguistics, 7(2), 149-182.
Rohdenburg, G. (1998) Clausal complementation and cognitive complexity in English. Paper presented at the Anglistentag, Erfurt, Germany.
Sedivy, J.C. (2002) Invoking Discourse-Based Contrast Sets and Resolving Syntactic Ambiguities. Journal of Memory and Language, 46, 341-370
Sedivy, J.C., Tanenhaus, M.K., Chambers, G.C. & Carlson, G.N. (1999) Achieving incremental semantic interpretation through contextual representation. Cognition, 71, 109-147.
Spivey-Knowlton, M. & Tanenhaus, M.K. (1994) Referential context and syntactic ambiguity resolution. In Clifton, C., L. Frazier, and K. Rayner (Eds.), Perspectives in Sentence Processing, Lawrence Erlbaum, 415-439.