Memorandum on Robert Smith
[ca. 11] April 1811
Having seen in the Aurora of the 5th. inst: & since copied into other Gazettes, an explanation which the Editor says he was authorized to make “of the rupture which has taken place between Mr. Madison, and Mr. R. Smith”1 I have thought it proper, whilst the circumstances are fresh in remembrance, to preserve them in the following memorandum:
On the 2 day of March Mr. S. called on me, as was common, on some point of official business. In the conversation, he alluded to the account in the newspapers, of the dismission of Mr. Pickering by Mr. Adams, as just published for the first time by the former.3 Altho’ the manner of Mr. S. did not denote any purpose beyond the ordinary conversation incident to such a topic, it happened to be the very day on which I meant to have sent for him in order to communicate the necessity of making a change in the head of the Department of State. Dropping therefore the case of Mr. Pickering, and breaking its apparent relation to his own, by the interposition of other subjects, I intimated that in coming over, he had anticipated my intention of sending for him, with a view to a conversation which would be as candid & explicit on my part, as it was in some respects delicate and disagreeable in itself. After remarking that I had delayed the execution of my purpose for some time after I had formed it, in order that my communication might have the character of being not the result of any sudden impulse, but of a deliberate regard to public considerations, and official duty, I proceeded to state to him, that it had long been felt, and had at length become notorious, that the Administration of the Executive Departmt. laboured under a want of the harmony & unity, which were equally essential to its energy and its success; that I did not refer to the evil as infecting our Cabinet consultations, where there had always been an apparent cordiality, and even a sufficient concurrence of opinion; but as shewing itself in language and conduct out of doors, counteracting what had been understood within to be the course of the Administration, and the interest of the Public; that truth obliged me to add, that this practice, as brought to my view, was exclusively chargeable on him; and that he had not only counteracted what had been the result of consultations apparently approved by himself, but had included myself in representations calculated to diminish confidence in the administration committed to me. He expressed surprize that I should have yielded to such impressions, declared that he had given no cause for them; observing that it was not to be conceived that a motive could be felt by him, to be otherwise than friendly personally, as well as to the credit of my administration. I told him that I had long resisted such impressions, well knowing that my conduct to him had merited a very different return. But that they were the result of facts and circumstances, brought to my knowledge, from so many sources and with so many corroborations, that it was impossible to shut my mind against them. I assured him that I had struggled agst. the belief as long as I could, that it was painful as well as difficult for me to suppose, that conscious as he must be of the friendship he had experienced in my nomination of him to the Department of State, and in the constant aids I had given him, in discharging its duties, he should privately set himself agst. me in any respect; but that what had harrassed my feelings in a degree equalled by no occurrence in a long political life, was the reflection that there were among those most nearly connected with him, a number of individuals whom I had always felt a gratification in classing among the best of my friends political & personal,4 & for whom I felt the highest esteem & the sincerest affection; and that the idea of distressing them was most severely so to myself. He repeated his solemn denial of unfriendly conduct in any way towards me, or having done any thing tending to obstruct or embarrass the administration: that on the contrary he had been always personally my friend, and had contributed as far as he could to the credit & support of the administration: What motive could he have to be otherwise, being himself a member of it, and having neither pretensions nor expectations of any higher sort? What could have given rise to the unfavorable sentiments I had expressed, he was at a loss even to conjecture. I told him I was aware of the awkwardness of my situation, in being obliged to refer to information and evidence which had come to me in ways not permitting me to name to him the sources; but I could assure him that the sources were such as made it my duty not to disregard them; and that unquestionably, he would himself, in my situation yield to the accumulated statements which had their effect on me. In what instances had he set himself agst. me, or against measures espoused by the Administration? I reminded him of a conversation with Mr. 5 reported by the latter, in which he had indulged himself in disparaging remarks on my official character, & that of others in the Cabinet; on the general course of my policy which he signified he disapproved; and in which he had communicated certain Cabinet proceedings, some of which were of so confidential a nature that the gentleman did not consider himself at liberty to repeat them. I had taken occasion before to drop him a hint that such a conversation had been given out; observing at the time, that I did it not because I lent an ear to it; but that it might suggest circumspection. He slighted then the report as proceeding from a source not likely to be listened to; and now repeated the denial of the conversation, with an allusion to a report from the same source, as to a conversation with another member of the Cabinet, where it appeared, that no interview could have taken place. I admitted that if this had been a solitary case, it would have been entirely dismissed from my recollection; but this was far from being the fact; altho’ I could not equally enter into a specification of other cases. For examples in which he had counteracted what he had not himself disapproved in the Cabinet, I referred to the Bills called Macon’s bills, and the non-intercourse bill; in the consultations on which he appeared to concur in their expediency; that he well knew the former, in its outline, at least, had originated in the difficulty of finding measures that would prevent what Congress had solemnly protested agst., towit, a compleat submission to the Belligerent Edicts; that the measure was considered as better than nothing, which seemed to be the alternative, and as part only of whatever else might in the progress of the business be found attainable; and that he neither objected to what was done in the Cabinet, (the time & place for the purpose); nor offered any thing in the place of it; yet it was well understood that his conversations & conduct out of doors, had been entirely of a counteracting nature; that it was generally believed that he was in an unfriendly disposition personally and officially; and that altho’ in conversations with different individuals, he might not hold the same unfavorable language, yet with those of a certain temper, it was no secret that he was very free in the use of it; and had gone so far as to avow a disapprobation of the whole policy of commercial restrictions, from the Embargo throughout. I intimated to him also that it was a complaint among our friends in Congs. that the Federalists frequently quoted him for communications from our Ministers abroad, which was unknown to others, the disclosures being sometimes such as to be deemed confidential, and to be turned agst. the administration.6 I glanced also at the report of his conversation with Mr. Morier, in which he (Mr. S) had expressed his disapprobation of the whole course of policy observed by the U. States towards G. B.7 All these facts he repelled by a repetition of what he had before said. With respect to his motives for dissatisfaction, I acknowledged that I had been, for the reasons given by him, much puzzled to divine any natural ones, without looking deeper into human nature than I was willing to do: and it was on this account that I had so long resisted the impressions which had at length been made on me; that instead of having any just motives to become an adversary, I knew, and he must be conscious, that in my confidential intercourse with him, in my kindness in general; and above all in the labor I had taken upon myself in behalf of his official duties, and for his credit, as well as that of the administration, I ought to have found an opposite return. On this subject as well as every other, I told him, I meant as I ought to be entirely frank, and must therefore say, that it was an imperious consideration for a change in the Departmt. of State, that whatever talents he might possess, he did not as he must have found by experience, possess those adapted to his station; that this had thrown the business more into my hands than was proper, or consistent with my own duties; that as long as I considered him in the light I once did, I had cheerfully given him my aid, but that it was too much to be expected under actual circumstances, and that moreover, the increase of the public business had put it out of my power to do his share as well as my own; and that indeed throughout it was not done as well as might have been, by a mind appropriated thereto. I observed that I could appeal to himself for the fact that the business of the Dept. had not been conducted in the systematic and punctual manner, which was necessary, particularly in the foreign correspondence, and that I had become daily more dissatisfied with it. He did not admit that complaint was well founded; intimating that I had a particular way of thinking on this subject, and that his conduct of the business would fully justify itself on examination. I told him he could not but be in a great error; reminding him of the condition in which his correspondence, more particularly, was brought to me; which was almost always so crude & inadequate, that I was in the more important cases generally obliged to write them anew myself, under the disadvantage sometimes of retaining, thro’ delicacy some mixture of his draft; that he must recollect that in the cases of Erskine & Jackson, the correspondence on his part had in a manner, fallen entirely on my hands.8 I reminded him also of important failures to make seasonable communications to our foreign Agents; particularizing the case of neglecting, tho’ repeatedly desired, to make known to our Minister at Paris, as was done to our Minister at London, that in case the Letter of the Duke de Cadore of Aug. 5. to Genl. Armstrong as reaching us through English newspapers, should it be officially confirmed, it would be the ground of a Proclamation as authorized by the Act of May; 1810. and the case of not keeping Mr. Shaler at the Havanna, duly informed of the state of our foreign relations, in consequence of which, as appeared by Mr. Shaler’s letters he was unable to pursue the object of his mission with advantage.9 I observed that if he had transmitted at once, in multiplied copies, & thro’ different channels, the same information for the French Govt. as to the B. Govt. as to the light in which the letter of the D. de Cadore was viewed, it might, by removing uncertainty & distrust as to the course here, have prevented the delay & embarrasment resulting from the course there. The impression made by these remarks was shewn rather by his manner, than his comment which was limited to a general disclaimer of the justness of them; & to allusions to a report that he had expressed to Mr. 10 Ingersoll, lately in Washington, a disapprobation of the Proclamation putting in force the non-importation act agst. G. B. which he denied to be fact, & said that he had sought out that Gentleman, and had obtained from him a satisfactory explanation. In this stage of the conversation, but in what particular connection is not recollected, it was noticed as a mark of his disinclination to co-operate in promoting measures for the better fulfilling the Executive trust, that altho’ the Act of Congs. at the Session preceding that just closed, relating to our diplomatic establishment, & of course particularly affecting his dept. had been found so very inconvenient, and it had been so often suggested to him; as desireable that some active member of Congress, should be apprized of the expediency of amending or repealing the Act, yet no such hint had ever been given, till at length I had availed myself of an opportunity of explaining the matter to a member of the Senate, who readily introduced it to the Senate, but too late in the session, to receive an effectual attention.11 He signified that he had not been in the habit of proceeding in such a way with business belonging to the Legislature: and seemed to disapprove or doubt the propriety of it. I remarked that where the intention was honest & the object useful, the conveniency of facilitating business in that way was so obvious that it had been practised under every past administration, & wd. be so under every future one; that Executive experience wd. frequently furnish hints & lights for the Legislature; that nothing was more common than for members of Congs. to apply for them; and that in fact, such communications, in cases not calling for formal messages, were indispensible to the advantageous conduct of the public business. A resort to formal messages on every occasion where executive information might be useful, was liable to obvious objections. He made no particular reply; but did not seem to acquiesce. Returning to the necessity of harmony & unity in the Executive Councils, in providing for which, I expressed a disposition to wound feelings any where as little as possible, he said he had himself regretted my situation, in reference to the want of cordiality among members of the Cabinet, declaring at the same time, that whilst he was aware of intrigues & hostilities carried on agst. himself, he had abstained from every thing of that sort agst. others, disdaining at all times to stoop to such practices. I told him it was unnecessary to repeat observations which I had already made; that such was the state of things that a remedy had become essential in the view of the most considerate friends of the Administration, and that I wished for the reasons given, to make it as lenient as would answer the purpose. It had occurred to me that he might not be disinclined to serve his Country in a foreign mission; and that St. Petersburg, where there was a vacancy, might be an eligible as it certainly was an important situation. London more so, he remarked quickly. For London, I replied, another arrangement was thought of;12 adding, with a view to repress miscalculations, that it was a place of discussions & negociations calling for appropriate talents & habits of business. He said he had for a considerable time entertained thoughts of retiring from the Department of State, and had looked towards a vacancy on the Bench of the supreme Court, likely to be produced ere long, by a death in Baltimore (alluding to Judge Chase).13 I observed that in that event, it might be found most proper to seek a successor elsewhere, intimating also that he had been long out of the practice & study of the law, and that the Senate would probably be hard to please in such a case. He made light of that consideration; with an expression of confidence in his standing there, which led me to remark that he was not aware how much room there was for a different estimate, that he had assuredly lost ground extremely with the members of both Houses of Congress, insomuch that the prevailing sentiment, as brought to my knowledge in the most direct manner, and from some quarters not unfriendly to himself, called for some arrangement that would at least vary the composition of the Cabinet.14 He ascribed unfavorable impressions agst. him as far as they might exist to intrigues & calumnies; signifying that there was however a body of firm friends personal & political, who would not desert him whatever course things might take. I did not admit that any considerable body of the Republicans, would in any event take side agst. the Administration; that on the contrary, many on whom he might perhaps count, had become dissatisfied with the course he had pursued; that it was not so much therefore the consideration alluded to by him, which weighed with me, tho’ not without weight especially at the present crisis in Maryland (the approaching election of Senatorial Electors),15 as the one I had before mentioned, namely the personal friends common to both of us, that made me desirous of smoothing the change become necessary, by proposing a mission to Russia, which I sincerely wished him to accept. I remarked that the services there tho’ neither difficult nor laborious, might be important; that the station was respectable, and that it was desireable to find a minister whose political grade here had been such as would satisfy the expectations of the Emperor, and whose private resources would also aid his salary in bearing the expensiveness of that Metropolis & Court. He admitted an inclination towards a trip to Europe as more eligible than his situation here; and after a few uninteresting observations, concurred in the measure with a mutual understanding that the appointment would be postponed for some days, till he could wind up the business of his Department, and prepare for his departure from Washington. I observed that as the 1st. of April, closed a quarter it might be a convenient epoch, for the date of his Commission; in which he acquiesced. He said he supposed there would be no impropriety in letting it be known that the mission was on foot—None at all—After a short pause, May I say that the appointment is offerred to me. I have no objection, it being of course understood that it is to take place on the 1st. of April; and that you will let me be at liberty as many days previous as may be convenient, to take overt measures for supplying the vacancy; which he promised. The conversation closed with his proposal that it should be considered as entirely confidential, & my acquiescence in it.
From his conversations & conduct, for several days, in his office & elsewhere, it was not doubted that he persisted in his intention to accept the Mission, and was making preparations accordingly. Circumstances soon however began to denote & strengthen doubts, particularly his declining, after accepting my invitation to dine with a party including the Russian Legation, and as I did not hear from him as was expected and the 1st. of april approached, I sent for him.
On his arrival, I told him my object, and that I had, according to the understanding between us, caused a Commission to be made out for him. He said he was himself on the point of coming over to me,16 with the view of returning into my hands, his Commission of Secretary of State, (handing it to me at the same time), and to inform me that he had determined to decline the other which had been proffered to him. However disposed he might have been to accept it under other circumstances, it was impossible he could do so under such as would give it the appearance of a mere expedient to get rid of him as Secretary of State. He had learned from Baltimore that a removal of him was believed to have been determined on, under the influence of intrigues agst. him, and that this intention was known even to federal members of Congress, as was evinced by their language on their return home; that the same impression existed elsewhere; that he had in fact, recd. letters from his friends not only in Baltimore, but in Penna. & N. York,17 advising him by no means to make himself a party to the transaction, by accepting the Russian mission, which would be regarded as a mere cover for his removal. I told him I could not be answerable for the reports or assertions that might be propagated; that the course I had pursued was the one deemed proper in the circumstances which had resulted from that pursued by him; and had been as delicate and favorable to him, as could be reconciled with what I owed to the public & to myself; that in tendering him the Commission for Russia, I wished him to accept it for the reasons explained to him; that what the Federalists said on the occasion, must have grown out of the conversations which had, as was well known, been frequent & free among the friends of the Administration, on the necessity of a change in the Department of State. I availed myself of this turn of the conversation, to allude anew to the reports & complaints, that the Federalists were the first to get from him information of our foreign Affairs; and to its being understood that he had told Mr. Morier that the whole policy of the government towards G. B. had been contrary to his Opinion & advice. This he denied. I assured him there was full evidence that Morier had said so; that this was known to and believed by sundry members of Congs. and had contributed, with other causes, to strengthen the current running agst. him. I reminded him of the official letter from Mr. Morier to him complaining of the non-intercourse being enforced against G. B. during the actual conduct of France in which he (M) referred to a conversation in which he (S.) admitted that G. B. had a right to complain; I told him I had been surprized when he communicated the letter to me, to find no apparent intention of a formal disavowal of that circumstance; till I had pressed it on him as material to himself in case the correspondence should be brought before the public or Congress; and that I did not approve of the course, finally taken by him, of getting Morier to withdraw the letter and substitute another omitting the passage; a course less eligible than the one I had suggested, of a written disavowal, as Morier’s communications to his Govt might correspond with his first letter, and might find their way to the public thro a Call for papers, by the British Parliament, in which case the statement would be without his contradiction.18 These I observed were disagreeable topics, and I willingly turned from them, to repeat to him, that with a wish to consult the sensibility of common friends, I had been ready to give him in exchange for an office which he professed, no longer to relish, a foreign Mission which in itself did not appear to be unacceptable to him; and that it was still in his option, & would remain so for a short time longer, if he wished to deliberate further on the subject. He said he had made up his mind, & meant to be understood as having given his final answer to the proposal. He recurred to the aspect it wore of an indirect removal of him, from the department of State, and to the allegation of intrigues agst. him, which had been mistaken for a loss of Confidence with the public, & with Congs.; regretted the tendency of what was taking place to injure the Republican cause, observing again that he should be supported by a Body of friends, and that he knew he could stand on good ground in justifying himself to his Country. I assured him that neither my sentiments nor conduct in relation to him were in the least the effect of intrigues to which I should never listen, but of the facts & considerations I had unfolded to him; that I did not doubt the friendship for him of a number of respectable & weighty characters, but it was not less true, however disagreeable it might be to dwell on the circumstance, that with the public as well as among the members of Congs. in both House[s], the tide was setting strongly & extensively agst. him, that I regretted as much as himself a tendency in any occurrence to impair harmony among the Republicans, more especially at this time & in this State, but that I believed this was not likely to be much the case; conceiving that the administration rested on ground as solid as at any preceding period; & that for myself, I was entirely confident that what I had done in relation to him, could be justified not only to the public, if it should become there necessary, but even to the most partial of his personal friends; that I cd. have no personal objection therefore to any step he might take which would call the public attention to it. He said it was not his wish, however confident he might be of the ground on which he stood, to introduce any public discussion.19 The conversation being at an end, he took his leave with a cold formality, and I did not see him afterwards.
On reading over the above, I recollect nothing worth mentioning which is omitted; unless it be thought an exception, that in some stage of the conversation, I alluded to the pretty general opposition made by his brother in the Senate to the measures proposed, or supposed to be approved, by the Executive, and its effect in strengthening the presumption with many, of a like spirit in the Secretary of State; explicitly declaring at the same time, that however I might be sometimes disappointed at the part taken by his brother, or regret it on account of his talents & his weight, I had always considered myself bound to suppose him actuated by a just respect for the independence of his station & his character; and that as he stood in no official connection with the Executive rendering him any wise responsible for his political conduct, I had never permitted myself to complain of it.
Ms (DLC). In JM’s hand. At a later time JM wrote on the cover: “(quere—if necesary to become public) / Memorandum as to R. Smith / Apl. 1811.”
1. The 5 Apr. 1811 issue of the Philadelphia Aurora General Advertiser announced that the editor was “authorized” to state that the rupture between JM and Robert Smith arose from conflicts over four issues. The first was the two Macon bills and the Nonintercourse Act; the second was “certain nominations” JM had made to the Senate; the third was the practice of recommending measures “in secret, and individually, to members of congress, instead of the fair, unembarrassing, and open course of public recommendation”; and the fourth was the pecuniary transactions of “a certain public agent,” which, the editor predicted, would probably come before the next session of Congress. Other grounds for the rupture included “Gallatin and his conduct with Erskine,” which the Aurora had discussed in the past and, the editor promised, would do so again in two pieces already prepared for the press. The editorial page also reported that Robert Smith had been offered “banishment to Siberia, but has declined.”
2. Left blank by JM.
3. On 19 Mar. 1811 the National Intelligencer published five letters exchanged by John Adams and Timothy Pickering between 10 and 12 May 1800 relating to the dismissal of the latter as secretary of state. The next issue, on 21 Mar., announced that Robert Smith had been offered the mission to the court of St. Petersburg.
4. JM was probably alluding to the fact that the Smith family of Baltimore, on three occasions, had married into the Nicholas family of Virginia. George Nicholas married Mary Smith, sister of Robert and Samuel Smith; Philip Norborne Nicholas married Mary Spear, sister of the wife of Samuel Smith; and Wilson Cary Nicholas married Margaret Smith, another sister of Robert and Samuel Smith.
5. Left blank by JM.
6. See, for example, John Graham to JM, 1 Sept. 1809, and George W. Erving to JM, 28 Nov. 1809 (PJM-PS description begins Robert A. Rutland et al., eds., The Papers of James Madison: Presidential Series (3 vols. to date; Charlottesville, Va., 1984—). description ends , 1:348–49 and n. 5, 2:88 and n. 5).
8. See Madison, Francis James Jackson, and Robert Smith, 9 Oct.–11 Nov. 1809, PJM-PS description begins Robert A. Rutland et al., eds., The Papers of James Madison: Presidential Series (3 vols. to date; Charlottesville, Va., 1984—). description ends , 2:8–11.
9. In pursuing his instructions “to feel the pulse of Cuba as to an estimate of the inducements to an incorporation of that island with the United States,” Shaler had held conversations with Cubans about the consequences for them of an Anglo-American war. The conversations, he reported, were “necessarily general” as he was “unable to give any specific information on the state of our foreign relations and the news lately from home being very uncertain” (Shaler to Robert Smith, 3 Feb. and 4 Mar. 1811 [DNA: RG 59, CD, Havana]; on Shaler’s mission, see PJM-PS description begins Robert A. Rutland et al., eds., The Papers of James Madison: Presidential Series (3 vols. to date; Charlottesville, Va., 1984—). description ends , 2:310–12).
10. Left blank by JM.
11. JM was referring to his disagreement with Smith over the bill to “fix the compensation of public ministers, and of Consuls residing on the coast of Barbary,” passed by Congress on 1 May 1810 (see JM to the House of Representatives, 27 Apr. 1810, PJM-PS description begins Robert A. Rutland et al., eds., The Papers of James Madison: Presidential Series (3 vols. to date; Charlottesville, Va., 1984—). description ends , 2:326 and n. 1).
12. In their reports to their governments that the “schism” in the cabinet between JM and Smith had broken out publicly, both Louis Sérurier and John Philip Morier mentioned that Secretary of War Eustis would replace William Pinkney in London (Sérurier to Cadore, 9 Apr. 1811 [AAE: Political Correspondence, U.S., vol. 65]; Morier to Wellesley, 26 Mar. 1811 [PRO: Foreign Office, ser. 5, vol. 74]). Rumors to this effect circulated for some time, and according to William Duane “both Mr. Madison and Mr Gallatin did endeavor to get rid of Dr Eustis by a proposition to send him ambassador to England.” The proposition, Duane added, “was made in the Cabinet—that Mr Munroe moved to send an ambassador—and that Mr Gallatin said ‘between the wind and water’—if one is to be sent, ‘he should be from the Eastern states—a man of the highest standing—and in his opinion from Massachusetts.’ It was not agreed to nor did the doctor take the hint” (Duane to Henry Dearborn, 9 Jan. 1812 [CSmH]).
13. Supreme Court Justice Samuel Chase died on 19 June 1811. JM replaced him with Gabriel Duvall of Maryland (National Intelligencer, 22 June 1811; Senate Exec. Proceedings description begins Journal of the Executive Proceedings of the Senate of the United States of America (3 vols.; Washington, 1828). description ends , 2:189).
15. JM referred to the elections for the Maryland state senate, to be held on 2 Sept. 1811 (National Intelligencer, 3, 5, and 7 Sept. 1811).
16. On 29 Mar. 1811 the Baltimore Whig printed a “rumor” that Smith would resign that day as secretary of state.
17. In his diary Joseph Gales, Jr., recorded that Smith declined the mission to Russia, “as it was rumored and believed, by the advice of General Armstrong” (“Recollections of the Civil History of the War of 1812,” Historical Magazine, 3d ser., 3 [1874–75]: 161).
18. Morier’s dispatches to Lord Wellesley contain no material that explicitly confirmed JM’s fears on this score. However, the British chargé did report instances where Smith’s conversations, if made public in the manner JM anticipated, had the potential to embarrass the administration (see Morier to Wellesley, 24 Jan. and 22 Feb. 1811 [PRO: Foreign Office, ser. 5, vol. 74]).
19. After leaving Washington on 6 Apr. 1811 Robert Smith returned to Baltimore, where he complained to one of his correspondents that “among the means employed to mislead the enfeeble⟨d⟩ mind of our panic-struck President he was told that Armstrong & I had at Washn. form⟨ed⟩ a plan to Oppose him at the ensuing Election!!” (Robert Smith to John Bullus, [8?] Apr. 1811 [MdHi]). He then spent the next few weeks compiling an attack on JM, which he seems to have completed by 7 June and published in Baltimore three weeks later in both newspaper and pamphlet form as Robert Smith’s Address to the People of the United States.